-- Logs begin at Tue 2022-09-27 15:41:39 UTC, end at Tue 2022-09-27 16:55:22 UTC. -- Sep 27 16:15:44.232902 dv-cinder-855060-2-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Sep 27 16:15:44.242257 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: *** Starting uWSGI 2.0.18-debian (64bit) on [Tue Sep 27 16:15:44 2022] *** Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: nodename: dv-cinder-855060-2-iscsi Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: machine: x86_64 Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: clock source: unix Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: pcre jit disabled Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: detected number of CPU cores: 4 Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: current working directory: / Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: detected binary path: /usr/bin/uwsgi-core Sep 27 16:15:44.244843 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: your processes number limit is 47680 Sep 27 16:15:44.245494 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: your memory page size is 4096 bytes Sep 27 16:15:44.245494 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: detected max file descriptor number: 2048 Sep 27 16:15:44.245494 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: lock engine: pthread robust mutexes Sep 27 16:15:44.245494 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: thunder lock: enabled Sep 27 16:15:44.245494 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Sep 27 16:15:44.245494 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Sep 27 16:15:44.285802 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: Python main interpreter initialized at 0x55dba4e529c0 Sep 27 16:15:44.285802 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: python threads support enabled Sep 27 16:15:44.285802 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: your server socket listen backlog is limited to 100 connections Sep 27 16:15:44.285802 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: your mercy for graceful operations on workers is 90 seconds Sep 27 16:15:44.286143 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: mapped 403077 bytes (393 KB) for 2 cores Sep 27 16:15:44.286225 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: *** Operational MODE: preforking *** Sep 27 16:15:44.286480 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: *** uWSGI is running in multiple interpreter mode *** Sep 27 16:15:44.286480 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: spawned uWSGI master process (pid: 81601) Sep 27 16:15:44.286670 dv-cinder-855060-2-iscsi systemd[1]: Started Devstack devstack@keystone.service. Sep 27 16:15:44.288229 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: spawned uWSGI worker 1 (pid: 81612, cores: 1) Sep 27 16:15:44.288229 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: spawned uWSGI worker 2 (pid: 81613, cores: 1) Sep 27 16:15:44.288229 dv-cinder-855060-2-iscsi devstack@keystone.service[81601]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Sep 27 16:15:48.636866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 16:15:48.636866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: from cryptography.utils import int_from_bytes Sep 27 16:15:48.637718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 16:15:48.637718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: from cryptography.utils import int_from_bytes Sep 27 16:15:48.638414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 16:15:48.638414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: from cryptography.utils import int_from_bytes Sep 27 16:15:48.638880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Sep 27 16:15:48.638880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: from cryptography.utils import int_from_bytes Sep 27 16:15:48.691554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 27 16:15:48.691739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Sep 27 16:15:48.691739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 27 16:15:48.691908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Sep 27 16:15:48.691987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] command line args: [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 27 16:15:48.692061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] command line args: [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Sep 27 16:15:48.692182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 27 16:15:48.692272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Sep 27 16:15:48.692344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ================================================================================ {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 27 16:15:48.692412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ================================================================================ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Sep 27 16:15:48.692604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] admin_token = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.692736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] admin_token = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.692851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config_dir = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.692958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] config_dir = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] config_source = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] config_source = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] debug = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] debug = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.693957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] insecure_debug = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.694853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_config_append = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_config_append = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_dir = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_dir = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_file = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.701968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.702726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.703941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] notification_format = cadf {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.64/identity {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.64/identity {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] publish_errors = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] publish_errors = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.704994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] standard_threads = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] standard_threads = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] strict_password_check = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.705995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] use_eventlog = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_journal = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] use_journal = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_json = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] use_json = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_stderr = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] use_stderr = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] use_syslog = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] use_syslog = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] watch_log_file = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Sep 27 16:15:48.706821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.706892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.706963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.707938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.708951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.709890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.710908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.711957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.712954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.713957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.714998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.715935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.716860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.717996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.718963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.719960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.720947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.enabled = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.721909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_password = {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.722919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_sasl_enabled = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.723854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.memcache_username = {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.724974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725301 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.725914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.726948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.727912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.external = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.728931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.external = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.mapped = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.password = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] auth.token = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.729884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.password = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] auth.token = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.730826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] catalog.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.731997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.732987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.733948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.734974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735237 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.735973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.736971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.737977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.738965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.739128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.739205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.739374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.742926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.743957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.744977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.745987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.746764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.747427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.747608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.747690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.747853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.747970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.password = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.748943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.749892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.750876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.751987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.752975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.753925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.754980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.755967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.756922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.757877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] policy.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.758843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.759971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.760934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.761993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.762153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.762399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.762476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.762693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.763445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.763535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.763610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.763680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] role.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.763791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.763923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] role.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] role.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.764912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.765910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.766925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.767919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.768891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.769879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.770844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.771995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.772943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.provider = fernet {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.773994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] trust.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.774916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.775999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.776916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.connection = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.777960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.778943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.779928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.780930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.781089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.781215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.781293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.781373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Sep 27 16:15:48.781446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81613) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 27 16:15:48.781516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=81612) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Sep 27 16:15:48.811330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.811556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.811780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.811912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.812146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.812369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.820213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.820736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.849029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.849219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.849310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.849452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.906138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.906309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.956692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.956916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.957264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:48.957754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.128794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.129938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.130069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.130282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.130493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.153367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.153594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.153820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.153908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.154930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.155131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.165893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.166143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.187997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81613) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.188847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=81612) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Sep 27 16:15:49.210448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.210617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.210710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 16:15:49.210850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 16:15:49.211011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.211193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.211289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 16:15:49.211394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Sep 27 16:15:49.211477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.211666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.211748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.211925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.212014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.212181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.212263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.212437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.221925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.222206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.222293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.222480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.222563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.222750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.222854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.223057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.229704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.229971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `type` (API Prefix: ) [/credentials, /credentials/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.233132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.233415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.233513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `type` (API Prefix: ) [/domains, /domains/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.233700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.233783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.233984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.234920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.235975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.236161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.258997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.259423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.260465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.260748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.260959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `type` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.261244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.265423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.265708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.265922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `type` (API Prefix: ) [/groups, /groups/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.266025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.266215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.266496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.272315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.272598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.272801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `type` (API Prefix: ) [/limits, /limits/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.273087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.278726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.279054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.279344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.279429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `type` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.279635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.279728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.279915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.280822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.281010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.281277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.296646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.296925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.297500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.297778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.300734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.301612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `type` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.306857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.306857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.306999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.306999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.308319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.309295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `type` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.311496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.312582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `type` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.314701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.315027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.315317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.315602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.315816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.315901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.316116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.316199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.316400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.316686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.316962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.317253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.331762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.332058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.332576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.332853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.333372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `type` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.333664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.333932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.334200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.339722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.341642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH2: ['/token' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.341837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.343208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.343392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.343671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.344731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.345006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.346391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.346679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.346996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.347285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.347683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.347875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `type` (API Prefix: ) [/policies, /policies/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.348158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.348437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.348728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.349121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.359384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.359674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.359963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.360254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.360637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.360933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.361033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `type` (API Prefix: ) [/projects, /projects/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.361195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.361286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.361561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.361841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.362217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.362502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.362806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.377297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.378846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `type` (API Prefix: ) [/regions, /regions/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.380227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.381767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `type` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.383114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.384557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.384642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.386033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.386165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.386313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.386719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.388223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `type` (API Prefix: ) [/roles, /roles/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.391872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.391872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.393704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.395187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.395357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.396814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `type` (API Prefix: ) [/services, /services/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.398059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.398459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.398748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.399056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.399733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.400128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.400413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.400688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.407086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.407381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.407667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.408600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `type` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.408892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.409172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.415008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81613) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.415321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.415622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.415947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.416045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `type` (API Prefix: ) [/users, /users/] {{(pid=81612) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Sep 27 16:15:49.416180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.416308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.416455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.416587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.416731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.416860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.417940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.418133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.418215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.418449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.418539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.418726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.418881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81613) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.419215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.419541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.419782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=81612) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Sep 27 16:15:49.453686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55dba4e529c0 pid: 81613 (default app) Sep 27 16:15:49.454978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WSGI app 0 (mountpoint='') ready in 5 seconds on interpreter 0x55dba4e529c0 pid: 81612 (default app) Sep 27 16:15:49.457068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ef435d-1af0-49de-a8b4-019ece2e6262 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:15:49.457619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ef435d-1af0-49de-a8b4-019ece2e6262 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:15:49.457896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ef435d-1af0-49de-a8b4-019ece2e6262 None None] PATH_INFO: `/v3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:15:49.462029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1/1] 10.222.0.64 () {50 vars in 791 bytes} [Tue Sep 27 16:15:49 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:15:57.156076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:15:57.156594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:15:57.156594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:15:57.160881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1/2] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:15:57 2022] GET /identity => generated 270 bytes in 9 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:15:57.166692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:15:57.167197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:15:57.167288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:15:57.239109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: INFO dbcounter [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] Registered counter for database keystone Sep 27 16:15:57.257697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG oslo_db.sqlalchemy.engines [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81613) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 27 16:15:57.264528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writer thread running {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 27 16:15:57.443553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG passlib.handlers.bcrypt [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81613) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 27 16:15:57.443925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG passlib.handlers.bcrypt [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81613) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 27 16:15:57.504945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:15:57.506662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:15:57.634952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 2/3] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:15:57 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 469 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 27 16:15:58.190973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:15:58.260879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: INFO dbcounter [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] Registered counter for database keystone Sep 27 16:15:58.282323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG oslo_db.sqlalchemy.engines [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=81612) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Sep 27 16:15:58.288596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writer thread running {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Sep 27 16:15:58.412797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c87eb0c-421b-40cf-a248-fd503a7076c6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:15:58.417101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:15:58.417736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:15:58.418017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:15:58.418286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:15:58.427484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:15:58.480343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING keystone.common.rbac_enforcer.enforcer [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 27 16:15:58.480642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb75e2d1-29f9-4278-883e-8f18fb66639a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:15:58.490697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 2/4] 10.222.0.64 () {58 vars in 1324 bytes} [Tue Sep 27 16:15:58 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 27 16:15:58.496796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:15:58.514813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a111144-b938-4f35-861a-9bf473c02607 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:15:58.519234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:15:58.520333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:15:58.520614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:15:58.520889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] PATH_INFO: `/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:15:58.522466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] RBAC: Authorizing `identity:get_project(project_id=9ac75ab2f7e84b088f06cdcdbcfae272)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:15:58.577152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING keystone.common.rbac_enforcer.enforcer [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Sep 27 16:15:58.577503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e918a837-053a-401e-a722-bcf24f39f3ac admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:15:58.582234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 3/5] 10.222.0.64 () {56 vars in 1207 bytes} [Tue Sep 27 16:15:58 2022] GET /identity/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272 => generated 344 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:00.769606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a2338-2d12-4bd9-8b17-8d33db7b582c None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:00.770130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a2338-2d12-4bd9-8b17-8d33db7b582c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:00.770130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d29a2338-2d12-4bd9-8b17-8d33db7b582c None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:00.774277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 3/6] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:00 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:00.779952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:00.780345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:00.780510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:00.800224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:00.801749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:00.809590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 4/7] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:00 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Sep 27 16:16:01.092912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a2338-2d12-4bd9-8b17-8d33db7b582c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:01.104842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d29a2338-2d12-4bd9-8b17-8d33db7b582c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:01.109681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a5318d-bd2f-4033-ab09-7e4711d51398 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:01.110277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a5318d-bd2f-4033-ab09-7e4711d51398 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:01.110559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a5318d-bd2f-4033-ab09-7e4711d51398 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:01.110868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a5318d-bd2f-4033-ab09-7e4711d51398 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:01.118374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a5318d-bd2f-4033-ab09-7e4711d51398 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:01.120183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a5318d-bd2f-4033-ab09-7e4711d51398 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:01.129267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 4/8] 10.222.0.64 () {58 vars in 1324 bytes} [Tue Sep 27 16:16:01 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:16:01.134538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:01.144009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c0e3ce-146b-4b90-a148-d86862d03ee7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:01.148297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d5ff6e1-7094-430b-8979-f785d4e6325f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:01.148913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ff6e1-7094-430b-8979-f785d4e6325f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:01.149194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ff6e1-7094-430b-8979-f785d4e6325f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:01.149462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d5ff6e1-7094-430b-8979-f785d4e6325f admin admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:01.150756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ff6e1-7094-430b-8979-f785d4e6325f admin admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:01.152607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d5ff6e1-7094-430b-8979-f785d4e6325f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:01.154730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 5/9] 10.222.0.64 () {56 vars in 1198 bytes} [Tue Sep 27 16:16:01 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 21 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:03.364779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859673a3-45d0-4b1c-94d7-b562173775ab None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:03.365293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859673a3-45d0-4b1c-94d7-b562173775ab None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:03.365387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-859673a3-45d0-4b1c-94d7-b562173775ab None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:03.366628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 5/10] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:03.371801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1263a056-336e-4d63-867b-752e24dda04c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:03.372881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1263a056-336e-4d63-867b-752e24dda04c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:03.373152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1263a056-336e-4d63-867b-752e24dda04c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:03.391060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1263a056-336e-4d63-867b-752e24dda04c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:03.392253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1263a056-336e-4d63-867b-752e24dda04c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:03.421101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fed2c91-26a3-411a-941f-fde05c868e9f None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:03.421388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fed2c91-26a3-411a-941f-fde05c868e9f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:03.421659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fed2c91-26a3-411a-941f-fde05c868e9f None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:03.423028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 6/11] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:03.427939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:03.428218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:03.428498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:03.471412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG passlib.handlers.bcrypt [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] detected 'bcrypt' backend, version '4.0.0' {{(pid=81612) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Sep 27 16:16:03.471802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG passlib.handlers.bcrypt [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=81612) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Sep 27 16:16:03.478924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 6/12] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:03.520673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:03.522272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:03.621227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 7/13] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:03.747801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1263a056-336e-4d63-867b-752e24dda04c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:03.772629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:03.852365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1263a056-336e-4d63-867b-752e24dda04c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:03.885654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:03.886225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:03.886506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:03.886794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:03.894406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:03.898515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:03.907576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:03.908513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:03.909431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 7/14] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:03.914400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:03.995059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cd6868c-b548-42a4-8bdf-69d5d0f4a76c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:03.997946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:03.999920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:04.018640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b90fbf2-cb32-454d-a939-bdd4c664fdb4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:04.032928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.033911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.034193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.034467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.053613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.054222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.054503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.054789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.055463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:04.055766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.057158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.069400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 8/15] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:04.075207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:04.101411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.102974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.175517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62ca3177-3fe0-43f7-b9a5-dbd5f6018794 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:04.201029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 8/16] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:16:04.206548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:04.209136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.209527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:04.209763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.210506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.210797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.211949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.213418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.214980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 9/17] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:04 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:04.221057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:04.235963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:04.304622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90ba740e-2616-4d87-b077-ffaeccd51c9c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:04.322887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f26054-d3be-446e-9413-7750772917f7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:04.343287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.343834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.344114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.344386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.358638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.359305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.359588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.359863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.360984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.362431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.362873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.364037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 10/18] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:04.364318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.369811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:16:04.383027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:04.384256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:16:04.385078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:16:04.385078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:16:04.385078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:04.385078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 9/19] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:04.390986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:04.445592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:04.451440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:04.470740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55ab4abb-49af-4ffa-944b-b7009701b965 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:04.494763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa57cd40-60ad-4d1e-9d01-785fbd17acf5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:04.504807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.505420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.505705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.505981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.506703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.domain.id=default, user.id=3c05ec42fc16441e83ec77e10313ea85) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:04.507043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.508505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f81def8-540b-4050-b817-25b5b0e8eb16 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.515926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 11/20] 10.222.0.64 () {56 vars in 1322 bytes} [Tue Sep 27 16:16:04 2022] GET /identity/v3/role_assignments?user.id=3c05ec42fc16441e83ec77e10313ea85&scope.domain.id=default&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 233 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:04.530204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:04.530888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:04.531173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:04.531449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:04.532116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:04.532429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:04.533889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae835361-b475-49f0-aee5-6d2bc1d24c72 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:04.555304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 10/21] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:16:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:06.840560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6100027-03d1-4f7d-adb8-83a0494897f2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:06.841098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6100027-03d1-4f7d-adb8-83a0494897f2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:06.841200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6100027-03d1-4f7d-adb8-83a0494897f2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:06.842555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 12/22] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:06.848036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cac525-bf14-405a-8a62-b4d16e3890ae None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:06.848468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cac525-bf14-405a-8a62-b4d16e3890ae None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:06.848633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cac525-bf14-405a-8a62-b4d16e3890ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:06.879813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-60cac525-bf14-405a-8a62-b4d16e3890ae None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:06.881054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-60cac525-bf14-405a-8a62-b4d16e3890ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:06.970521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 11/23] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:07.361704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee4148-5803-48b4-b66f-4244f829625f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:07.361985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee4148-5803-48b4-b66f-4244f829625f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:07.362439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ee4148-5803-48b4-b66f-4244f829625f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:07.363943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 13/24] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:07.369177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:07.369638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:07.370055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:07.394891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:07.396287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:07.456207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:07.482349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 12/25] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:07.519125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee4148-5803-48b4-b66f-4244f829625f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:07.672483 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ee4148-5803-48b4-b66f-4244f829625f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:07.714365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52096fe1-e7d0-475d-b446-4e4ef35642bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:07.715037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52096fe1-e7d0-475d-b446-4e4ef35642bf None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:07.715320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52096fe1-e7d0-475d-b446-4e4ef35642bf None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:07.715623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52096fe1-e7d0-475d-b446-4e4ef35642bf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:07.727559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52096fe1-e7d0-475d-b446-4e4ef35642bf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-52096fe1-e7d0-475d-b446-4e4ef35642bf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:07.733095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:07.733922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:07.734668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 14/26] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:07.743875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:07.761291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5fec13-ed26-48e3-8de3-ad7190beb0b5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:07.761584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5fec13-ed26-48e3-8de3-ad7190beb0b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:07.761857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b5fec13-ed26-48e3-8de3-ad7190beb0b5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:07.763866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 15/27] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:07.777537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:07.777801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:07.778107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:07.791238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:07.813756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:07.815025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:07.919738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0c536e1-b63a-40cd-b377-cb718c418384 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:07.935239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 16/28] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:07.935866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:07.996118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:07.997774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:07.998326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:07.998800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:07.999797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:08.000298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.001870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab8e7792-7ea6-40e6-b6b3-a70c55be44fa None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.023983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 13/29] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:08.025549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a10e91-e82f-4435-9d9b-03caba9de4b6 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.026022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a10e91-e82f-4435-9d9b-03caba9de4b6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.026489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a10e91-e82f-4435-9d9b-03caba9de4b6 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.027904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 14/30] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:08.029870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a10e91-e82f-4435-9d9b-03caba9de4b6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:08.072459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-785a9a27-e63e-4b1d-a7e2-ab0602ff1d64 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:08.109686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:08.110649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.111080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.111780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.119076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:08.128850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.129571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:08.130621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2cddcdcf-2c8f-4426-b582-904dea162a79 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:16:08.160521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:08.161630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:16:08.162612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:16:08.162612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:16:08.162612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:08.166525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 17/31] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:08.167890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.168234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.168514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.199206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a10e91-e82f-4435-9d9b-03caba9de4b6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:08.207896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:08.209177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:08.237866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:08.238832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.239274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.239683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.241610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.249299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.251005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 15/32] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:08 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:08.255652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:08.342444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 18/33] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:08.343555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:08.395474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e1ce09-517c-4173-85d9-330f88729733 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:08.407849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:08.461774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:08.462303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.462825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.463102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.463754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:08.464058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.466232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.492841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 16/34] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:16:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:08.493680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:08.549815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c4bd79-1178-46f0-9cd2-993e2a8c8aaf None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:08.589685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:08.621415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:08.622171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.622479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.622964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.627628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.629209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.630951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 19/35] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:08 2022] GET /identity/v3/domains/default => generated 199 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:08.632030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:08.638656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725c23ab-b3ac-4ced-a55c-caefce218c5d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:08.677801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:08.678665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.679142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.679568 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.680390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.681941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.768183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b416ebba-d56f-41fe-9f66-ccd061976ad9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:08.803415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:08.804025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:08.804308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:08.804582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:08.805227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:08.806689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:08.894017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 17/36] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Sep 27 16:16:08 2022] POST /identity/v3/roles => generated 225 bytes in 400 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:08.894984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:08.959180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:09.019110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d37965-eb83-4214-ae73-faa8bbc9815b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:09.062380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d54d49bd-4af8-4b77-b20a-1fea881742cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:09.066509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d49bd-4af8-4b77-b20a-1fea881742cf None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.066509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d49bd-4af8-4b77-b20a-1fea881742cf None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.066509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d54d49bd-4af8-4b77-b20a-1fea881742cf None admin] PATH_INFO: `/v3/domains/default/users/3c05ec42fc16441e83ec77e10313ea85/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.066509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54d49bd-4af8-4b77-b20a-1fea881742cf None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=3c05ec42fc16441e83ec77e10313ea85, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:09.068416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d54d49bd-4af8-4b77-b20a-1fea881742cf None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:09.161426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 20/37] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:16:08 2022] POST /identity/v3/projects => generated 301 bytes in 530 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:09.162395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:09.202098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:09.216477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 18/38] 10.222.0.64 () {58 vars in 1361 bytes} [Tue Sep 27 16:16:08 2022] PUT /identity/v3/domains/default/users/3c05ec42fc16441e83ec77e10313ea85/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:09.218331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06bd7b-8b4e-406f-8f10-7ea76357ab30 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.218701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06bd7b-8b4e-406f-8f10-7ea76357ab30 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.219017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c06bd7b-8b4e-406f-8f10-7ea76357ab30 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.220441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 19/39] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:09.221711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9dca40-0dab-4800-aec5-a3228cdd0d92 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.222001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9dca40-0dab-4800-aec5-a3228cdd0d92 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.222282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9dca40-0dab-4800-aec5-a3228cdd0d92 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.223812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 20/40] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:09.230308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ffd75-a669-4e0b-b9bd-1679cd2c6077 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.233255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ffd75-a669-4e0b-b9bd-1679cd2c6077 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.233255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b4ffd75-a669-4e0b-b9bd-1679cd2c6077 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.255451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8b4ffd75-a669-4e0b-b9bd-1679cd2c6077 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:09.257386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8b4ffd75-a669-4e0b-b9bd-1679cd2c6077 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:09.305538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75f5ceeb-af4a-45fd-a635-3d48677cca98 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:09.355476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 21/41] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:09.358176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.358475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.358759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.360322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:09.361653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.362353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.363141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.364185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:09.367256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:09.386063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:09.388071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:09.405111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:09.468567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 21/42] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Sep 27 16:16:09 2022] POST /identity/v3/roles => generated 219 bytes in 306 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:09.479953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 22/43] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:09.636493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:09.745874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:09.752732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4c6edb-415d-435e-86a7-7352690d0ae9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:09.765461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:09.788692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:09.789340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.789623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.789899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.791127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:09.792596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:09.794806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 22/44] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:09.800340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:09.871640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d78b977-a47b-4168-84c6-0fca28a6f336 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:09.906150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a282cf-fafe-424f-a43e-ef6da1cf9065 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:09.907582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8878f60a-f314-413c-89f4-249af667bb9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:09.908272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8878f60a-f314-413c-89f4-249af667bb9e None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.908628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8878f60a-f314-413c-89f4-249af667bb9e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.908976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8878f60a-f314-413c-89f4-249af667bb9e None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.909738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8878f60a-f314-413c-89f4-249af667bb9e None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:09.911241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8878f60a-f314-413c-89f4-249af667bb9e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:09.940551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000c8519-5432-4e36-8590-44703b764d41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:09.941168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c8519-5432-4e36-8590-44703b764d41 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:09.941452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c8519-5432-4e36-8590-44703b764d41 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:09.941728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000c8519-5432-4e36-8590-44703b764d41 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:09.942407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000c8519-5432-4e36-8590-44703b764d41 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:09.943905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000c8519-5432-4e36-8590-44703b764d41 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:09.998881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 23/45] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Sep 27 16:16:09 2022] POST /identity/v3/roles => generated 223 bytes in 234 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:16:10.063116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 23/46] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:16:09 2022] POST /identity/v3/projects => generated 312 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:11.488672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835db8cb-e742-4716-9982-e2dcaa0d48b6 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:11.489176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835db8cb-e742-4716-9982-e2dcaa0d48b6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:11.489256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835db8cb-e742-4716-9982-e2dcaa0d48b6 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:11.490606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 24/47] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:11.496030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:11.496404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:11.496590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:11.516084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:11.517641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:11.528917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:11.586344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 24/48] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:11.858711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835db8cb-e742-4716-9982-e2dcaa0d48b6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:11.867618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:11.960110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835db8cb-e742-4716-9982-e2dcaa0d48b6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:11.992801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:11.993423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:11.993700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:11.993970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.005246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:12.011164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:12.012130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:12.013318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:12.014357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 25/49] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:11 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:12.019931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:12.119586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3df9468c-31e4-4ee6-b425-556d2a9f5fbe None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:12.156802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:12.157419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.157699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.157974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.158637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:12.158973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:12.160381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12a4630-833d-4727-a7b1-a0b2cc1f51e6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:12.168898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 25/50] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:12.174426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:12.239135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:12.244178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8189b7-033c-49da-89ba-b9b0533f4b58 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.244458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8189b7-033c-49da-89ba-b9b0533f4b58 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.244788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8189b7-033c-49da-89ba-b9b0533f4b58 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.246085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 26/51] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:12.251338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.251629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.251910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.270131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:12.271367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:12.273340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:12.274823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4812c90f-2c5e-4691-a3e4-55725a961f27 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:12.307532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:12.308161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.308441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.308712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.309911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:12.311419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:12.312997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 26/52] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:12 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:12.318337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:12.340065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 27/53] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:12.523102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05acd5a3-fda3-41f8-9f1f-e74ee0a94990 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:12.557812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:12.558441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.558724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.559034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.560936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:12.562392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:12.563995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 27/54] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:12.569861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:12.616119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:12.618944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:12.670404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05cc15c4-cb9b-4514-adb5-a7b8716c3d7f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:12.706171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:12.707093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.707380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.707654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.708376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.domain.id=default, user.id=3c05ec42fc16441e83ec77e10313ea85) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:12.708684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:12.710125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:12.716808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 28/55] 10.222.0.64 () {56 vars in 1322 bytes} [Tue Sep 27 16:16:12 2022] GET /identity/v3/role_assignments?user.id=3c05ec42fc16441e83ec77e10313ea85&scope.domain.id=default&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 530 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:12.717087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d85fe31-55f2-477b-938f-eba7fa8a37a7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:12.752943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cfc749-d70a-4b0a-91f2-7f29bfde18d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:12.753592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfc749-d70a-4b0a-91f2-7f29bfde18d1 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.753873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfc749-d70a-4b0a-91f2-7f29bfde18d1 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.754144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cfc749-d70a-4b0a-91f2-7f29bfde18d1 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.755339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfc749-d70a-4b0a-91f2-7f29bfde18d1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:12.756811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cfc749-d70a-4b0a-91f2-7f29bfde18d1 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:12.758456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 28/56] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:12.764203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:12.767243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:12.873299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6727d184-1cad-49ae-b4c1-4f81a1b5f6bc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:12.909723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46f5ae5b-f223-4004-87be-3c994450bf45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:12.910333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f5ae5b-f223-4004-87be-3c994450bf45 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:12.910609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f5ae5b-f223-4004-87be-3c994450bf45 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:12.910933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46f5ae5b-f223-4004-87be-3c994450bf45 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:12.911589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f5ae5b-f223-4004-87be-3c994450bf45 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:12.913027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46f5ae5b-f223-4004-87be-3c994450bf45 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:12.991594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 29/57] 10.222.0.64 () {60 vars in 1140 bytes} [Tue Sep 27 16:16:12 2022] POST /identity/v3/projects => generated 298 bytes in 228 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:15.134298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a88ae5b-baf2-4416-8fcf-e685e1077c15 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:15.134871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a88ae5b-baf2-4416-8fcf-e685e1077c15 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:15.134871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a88ae5b-baf2-4416-8fcf-e685e1077c15 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:15.136205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 29/58] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:15.141525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:15.141933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:15.142090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:15.160349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:15.161535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:15.214453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:15.232598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 30/59] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:15.504762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a88ae5b-baf2-4416-8fcf-e685e1077c15 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:15.574040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:15.605513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a88ae5b-baf2-4416-8fcf-e685e1077c15 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:15.638078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6e53645-61f2-4834-ba9a-363d732d891e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:15.638700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e53645-61f2-4834-ba9a-363d732d891e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:15.639011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e53645-61f2-4834-ba9a-363d732d891e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:15.639287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6e53645-61f2-4834-ba9a-363d732d891e None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:15.640396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e53645-61f2-4834-ba9a-363d732d891e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:15.641848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6e53645-61f2-4834-ba9a-363d732d891e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:15.643437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 30/60] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:15 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:15.648958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:15.744867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6eefd55a-71fb-4faa-ba8b-6b829402a899 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:15.776651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e17e139-cf5d-43ce-8c36-6f2696ffc775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:15.777181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e17e139-cf5d-43ce-8c36-6f2696ffc775 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:15.777460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e17e139-cf5d-43ce-8c36-6f2696ffc775 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:15.778259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e17e139-cf5d-43ce-8c36-6f2696ffc775 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:15.779184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e17e139-cf5d-43ce-8c36-6f2696ffc775 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:15.780626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e17e139-cf5d-43ce-8c36-6f2696ffc775 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:15.877040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 31/61] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:15 2022] POST /identity/v3/users => generated 275 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:19.248067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393cbf7c-dae3-4c3b-9137-8980ba1a5495 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.248576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393cbf7c-dae3-4c3b-9137-8980ba1a5495 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.248648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393cbf7c-dae3-4c3b-9137-8980ba1a5495 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.250100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 31/62] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:19.255849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2431d5fc-2268-4353-ac79-9f126ce60779 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.256384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2431d5fc-2268-4353-ac79-9f126ce60779 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.256384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2431d5fc-2268-4353-ac79-9f126ce60779 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.266152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6d216-1a14-4675-8b27-cbdec7071c6f None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.266438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6d216-1a14-4675-8b27-cbdec7071c6f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.266719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c6d216-1a14-4675-8b27-cbdec7071c6f None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.268362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 32/63] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:19.273264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88304a-5319-4172-9cfa-c4e1035c5ca5 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.278970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88304a-5319-4172-9cfa-c4e1035c5ca5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.278970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e88304a-5319-4172-9cfa-c4e1035c5ca5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.299023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2431d5fc-2268-4353-ac79-9f126ce60779 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:19.300410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2431d5fc-2268-4353-ac79-9f126ce60779 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:19.307263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2e88304a-5319-4172-9cfa-c4e1035c5ca5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:19.308504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2e88304a-5319-4172-9cfa-c4e1035c5ca5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:19.372812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:19.420480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 33/64] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 27 16:16:19.433542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f5acd2-8078-4dcc-a943-a3a089655589 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.434473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f5acd2-8078-4dcc-a943-a3a089655589 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.434761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f5acd2-8078-4dcc-a943-a3a089655589 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.437409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 34/65] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:19.447581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0cd214-15ff-4c0d-a9bb-b32353354ede None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.448218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0cd214-15ff-4c0d-a9bb-b32353354ede None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.448702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb0cd214-15ff-4c0d-a9bb-b32353354ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.449149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 32/66] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:19.483798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-bb0cd214-15ff-4c0d-a9bb-b32353354ede None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:19.484999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bb0cd214-15ff-4c0d-a9bb-b32353354ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:19.580463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:19.586495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 35/67] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:19.708408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a691e76a-095f-4590-890b-e3db362b8e7f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.708723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a691e76a-095f-4590-890b-e3db362b8e7f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.708965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a691e76a-095f-4590-890b-e3db362b8e7f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.710311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 33/68] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:19.720970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.721258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.721539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.731006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a691e76a-095f-4590-890b-e3db362b8e7f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:19.745270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:19.746947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:19.842043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 36/69] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:19.843536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:19.882125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a691e76a-095f-4590-890b-e3db362b8e7f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:19.924284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:19.924922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:19.925199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:19.925472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:19.939561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:19.944799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:19.945718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 34/70] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:19 2022] GET /identity/v3/roles/admin => generated 83 bytes in 216 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:19.950822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:19.986160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad3a595f-21a0-465f-b764-c4cc082f4b5d None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:19.986880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.014386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.022689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.023627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.024121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.024579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.036183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:20.041808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:20.042745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:20.043572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:20.045781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 37/71] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:19 2022] GET /identity/v3/roles/member => generated 84 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:20.046839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.067045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f2b4ab-dbf7-4b61-b9a8-33c3397c9940 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.102397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.103070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.103357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.103636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.116582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:20.122157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:20.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:20.123776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 35/72] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:20.124397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.156132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a15ff57-94e4-446f-a313-a9bd44f92919 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.194481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.195122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.195401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.195670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.196460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:20.196763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.198158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4201ba-33cf-4500-92b6-3bead5259191 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.207626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 38/73] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.208854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18b8a90-d99b-45e3-b015-bcbf9e551668 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.209143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18b8a90-d99b-45e3-b015-bcbf9e551668 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.209423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a18b8a90-d99b-45e3-b015-bcbf9e551668 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.210636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 39/74] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:20 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:20.211369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18b8a90-d99b-45e3-b015-bcbf9e551668 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.233094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.241979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b524afe-427f-4e1e-9c6a-50a088c01783 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.262097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.276911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.278080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.278399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.278672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.279359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:20.279665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.281057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.289047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 36/75] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.289857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.314222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a18b8a90-d99b-45e3-b015-bcbf9e551668 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.348811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.349427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.349706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.349974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.350636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:20.350972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.352332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24182879-0804-4b20-a067-316b33bbfd8a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.360956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 40/76] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.362108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.362376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.362647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.379785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:20.380984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:20.393795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f3280b-b559-4d79-acf3-69d523d5a6f3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.427786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.428413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.428691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.428971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.439216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:20.444386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 41/77] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:20.444503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:20.445307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:20.446068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:20.446663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.446735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 37/78] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/roles/member => generated 84 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:20.446735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.467838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.497722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.555168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30c77c02-4b10-45f9-9de1-02a54741248d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.558500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695abc34-db57-4fd2-aeb1-f09e408d0587 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.591537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.592180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.592461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.592735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.598532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.599202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.599484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.599764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.600903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.602366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.604161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 42/79] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.605262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.609511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.611258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.614824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 38/80] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.614824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.690537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.722231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b43815b-77f8-4293-9f2b-d0b3d9b8f733 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.727990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a6553a-bfdb-4052-9e53-e443127ef7ad None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.739820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.757432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.758046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.758322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.758596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.759282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:20.759589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.760973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.763872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.764499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.764775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.765048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.766204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.768018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.769248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 43/81] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.770012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.771879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 39/82] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:20.772732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:20.870453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b252a6-bab2-4e59-80fc-766dd49d18fb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.877696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88bd0c80-9338-4312-afdc-3ff235c3235f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:20.905316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.905921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.906044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.906195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.906468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.911353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:20.916078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:20.916078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:20.916078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:20.951278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:20.971882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.973346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:20.981939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:20.983467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.071055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 44/83] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:21.076443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.081280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 40/84] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:20 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:21.082183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.113537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.150737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.172883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-852a015e-9140-470a-a9ed-ebdf03753349 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.185305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e436cb49-3823-40fa-b7f4-9686080ebf6a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.208994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.209625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.209912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.210190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.211328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.212774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.214344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 45/85] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:21.215721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.221869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.222486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.222767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.223066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.288698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.290151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.317527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b95e3f-fdfe-4ea6-b414-a2251ba6cf52 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.320582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.352031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.352663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.352945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.353225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.354335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.356437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.358031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 46/86] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:21.358906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.375331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.392544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 41/87] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:21.393549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.456914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7e17b1-22b7-43dc-bcbc-f71a0f1e3873 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.491201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.491829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.492112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.492388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.492623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5929d7c1-38f6-4605-b11c-73a7e760887d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.493499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.494985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.497485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 47/88] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:21.498238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.522393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.527104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.527740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.528105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.528381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.529498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.530970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.537470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 42/89] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:21.537470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.581799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.598801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3500b8aa-7646-4b39-b903-71ba7d25418d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.632648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f89876-e4b3-4986-99a2-5544f97e2fbd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.632876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.633482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.633763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.634039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.634687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.636161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.664994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.666115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.666115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.666439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.717013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 48/90] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:21 2022] POST /identity/v3/users => generated 289 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:21.718043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.740628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:21.743464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.744606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.833239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cdf8981-2973-4c1a-b636-05a888905ede None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:21.868913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 43/91] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:21.870208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.875290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:21.875960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:21.876241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:21.876512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:21.877778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:21.879294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:21.880962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 49/92] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:21.881809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:21.893505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:22.003354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:22.005031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320b02d3-8e8f-4984-91e3-13903e4ee9b2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:22.006681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8184e3a-905b-45d9-8e15-fc6b38e197c5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:22.050310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:22.050971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:22.051260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:22.051534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:22.052267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=3c05ec42fc16441e83ec77e10313ea85) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:22.052578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:22.054028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:22.058804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:22.059834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:22.060125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:22.060221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 44/93] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/role_assignments?user.id=3c05ec42fc16441e83ec77e10313ea85&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:22.060699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:22.061449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=e6f8c1e119d74e408654eed7e007c7f0) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:22.061767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:22.062035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:22.063316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:22.072604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 50/94] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:21 2022] GET /identity/v3/role_assignments?user.id=e6f8c1e119d74e408654eed7e007c7f0&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=1448af55b1904e1199e3954829e9815a => generated 259 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:22.073481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:22.166728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:22.208829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5024453c-61eb-4be2-8a86-125b450362df None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:22.254634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:22.256101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:22.258802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:22.259362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:22.260349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] RBAC: Adding query filter params (role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=e6f8c1e119d74e408654eed7e007c7f0) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:22.261652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:22.264252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:22.282437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 51/95] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:22 2022] GET /identity/v3/role_assignments?user.id=e6f8c1e119d74e408654eed7e007c7f0&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 259 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:22.314506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38902906-59ee-4be1-9ff8-a615a1a46093 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:22.362163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ff88c4-39cf-4c83-89d5-9cb9fb23c159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:22.362855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff88c4-39cf-4c83-89d5-9cb9fb23c159 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:22.363141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff88c4-39cf-4c83-89d5-9cb9fb23c159 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:22.363416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ff88c4-39cf-4c83-89d5-9cb9fb23c159 None admin] PATH_INFO: `/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:22.365788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff88c4-39cf-4c83-89d5-9cb9fb23c159 None admin] RBAC: Authorizing `identity:get_project(project_id=b23d49abd36d42b1a2f35c94fd0590f3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:22.367318 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ff88c4-39cf-4c83-89d5-9cb9fb23c159 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:22.369118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 45/96] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:22 2022] GET /identity/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3 => generated 312 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:22.376102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:22.493110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-064b261e-c2b8-435b-863e-546130535fb4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:22.568471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:22.569013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:22.569290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:22.569561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:22.570293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=b23d49abd36d42b1a2f35c94fd0590f3, user.id=e6f8c1e119d74e408654eed7e007c7f0) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:22.570600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:22.572180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52fa6de3-917c-4ac4-9358-92ce5afbfe1a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:22.574059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:22.579607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 52/97] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:22 2022] GET /identity/v3/role_assignments?user.id=e6f8c1e119d74e408654eed7e007c7f0&scope.project.id=b23d49abd36d42b1a2f35c94fd0590f3&role.id=1448af55b1904e1199e3954829e9815a => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:24.858516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c7da8b-b8be-4265-aa6d-0e835d29adeb None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:24.859066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c7da8b-b8be-4265-aa6d-0e835d29adeb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:24.859066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86c7da8b-b8be-4265-aa6d-0e835d29adeb None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:24.860417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 46/98] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:24.865783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f0347-c011-4985-a0f7-5f6e645032c5 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:24.866184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f0347-c011-4985-a0f7-5f6e645032c5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:24.866346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f0347-c011-4985-a0f7-5f6e645032c5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:24.899523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2e4f0347-c011-4985-a0f7-5f6e645032c5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:24.901009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2e4f0347-c011-4985-a0f7-5f6e645032c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:24.935935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c42297-5348-44b4-a3b0-bcfe39e47905 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:24.936560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c42297-5348-44b4-a3b0-bcfe39e47905 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:24.937039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c42297-5348-44b4-a3b0-bcfe39e47905 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:24.938674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 47/99] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:24.944358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74d773b-6cba-4e72-9b3f-1edcb9956434 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:24.944862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74d773b-6cba-4e72-9b3f-1edcb9956434 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:24.945337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f74d773b-6cba-4e72-9b3f-1edcb9956434 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:24.971609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f74d773b-6cba-4e72-9b3f-1edcb9956434 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:24.974995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f74d773b-6cba-4e72-9b3f-1edcb9956434 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:25.028587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:25.052111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 53/100] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:25.112181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 48/101] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:25.176937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55f9771-e721-4449-91fa-4b03f3ed92d2 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:25.177260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55f9771-e721-4449-91fa-4b03f3ed92d2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:25.177501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a55f9771-e721-4449-91fa-4b03f3ed92d2 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:25.178937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 54/102] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:25.197798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affca81b-81a4-4906-a462-d791a73658ae None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:25.198117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affca81b-81a4-4906-a462-d791a73658ae None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:25.198364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affca81b-81a4-4906-a462-d791a73658ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:25.228080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-affca81b-81a4-4906-a462-d791a73658ae None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:25.229278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-affca81b-81a4-4906-a462-d791a73658ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:25.296929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157bf768-a09c-4fc5-8fc7-19b043ea3f88 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:25.297259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157bf768-a09c-4fc5-8fc7-19b043ea3f88 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:25.297490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-157bf768-a09c-4fc5-8fc7-19b043ea3f88 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:25.298847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 55/103] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:25.304661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:25.304948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:25.305227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:25.337155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:25.339570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:25.344759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 49/104] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:25.445101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 56/105] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:25.597277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affca81b-81a4-4906-a462-d791a73658ae None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:25.705755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:25.742201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affca81b-81a4-4906-a462-d791a73658ae None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:25.754809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:25.791225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:25.807475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:25.808120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:25.808423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:25.808734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:25.823882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:25.829449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:25.830313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:25.831372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:25.842114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 50/106] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:25 2022] GET /identity/v3/roles/reader => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:25.842114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:25.844045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df52b931-5d8c-4b18-987a-307df3e21bf3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:25.884065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:25.885511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:25.886000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:25.887152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:25.891714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:25.893588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:25.895617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 57/107] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:25 2022] GET /identity/v3/domains/default => generated 199 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:25.896716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.007386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f81f5bb-f614-4433-967b-b8e33d0eb879 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.025936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d7005c-ffe8-4518-bdbd-e85880d4ae53 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.038897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.047382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.048050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.048333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.048612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.049286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:26.049614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.051013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.061513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 58/108] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:25 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:26.061513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.070405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.071079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.071366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.071639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.084562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:26.089879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:26.090724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:26.091557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 51/109] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:25 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:26.092180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.111313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.170299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071b02b8-0a90-4e47-b87d-ce69a29757b5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.200236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1860531d-e77f-493a-9bc1-f8a46af655f8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.205272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.206143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.206421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.206693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.207365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.208819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.234730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4153bb4e-e805-4782-bccf-47e682e46072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.235466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4153bb4e-e805-4782-bccf-47e682e46072 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.235682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4153bb4e-e805-4782-bccf-47e682e46072 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.235962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4153bb4e-e805-4782-bccf-47e682e46072 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.246570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4153bb4e-e805-4782-bccf-47e682e46072 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4153bb4e-e805-4782-bccf-47e682e46072 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:26.251840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:26.252835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:26.253859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 52/110] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:26.254530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14120b-a22f-44b2-88b6-2fff3cfd08f2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.254530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14120b-a22f-44b2-88b6-2fff3cfd08f2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.254721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec14120b-a22f-44b2-88b6-2fff3cfd08f2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.255819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 53/111] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:26.256520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec14120b-a22f-44b2-88b6-2fff3cfd08f2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.326884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.356764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec14120b-a22f-44b2-88b6-2fff3cfd08f2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.375091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 59/112] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:16:26 2022] POST /identity/v3/projects => generated 302 bytes in 314 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:26.375946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.400313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd7d24d-cc02-4046-bd38-68a5fa0abb3e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.401050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7d24d-cc02-4046-bd38-68a5fa0abb3e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.401342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7d24d-cc02-4046-bd38-68a5fa0abb3e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.401608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd7d24d-cc02-4046-bd38-68a5fa0abb3e None admin] PATH_INFO: `/v3/users/4d1affdb59974e16940870b99847401e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.420669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd7d24d-cc02-4046-bd38-68a5fa0abb3e None admin] RBAC: Authorizing `identity:get_user(user_id=4d1affdb59974e16940870b99847401e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.422367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bd7d24d-cc02-4046-bd38-68a5fa0abb3e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.423614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.424165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 54/113] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/users/4d1affdb59974e16940870b99847401e => generated 289 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:26.424165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41aa1643-a39d-4c2a-9f6c-80e158325df7 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.424165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41aa1643-a39d-4c2a-9f6c-80e158325df7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.424165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41aa1643-a39d-4c2a-9f6c-80e158325df7 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.424165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 55/114] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:26 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:26.432348 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41aa1643-a39d-4c2a-9f6c-80e158325df7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.497866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-313be512-283c-4d73-972c-b3dfa5ac24a4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.534117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.535516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.535934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.536153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.537001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:26.537311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.538734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22315671-1627-4ac5-b786-d507e9090eed None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.549319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 60/115] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:26.549488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41aa1643-a39d-4c2a-9f6c-80e158325df7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.552541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea71616-6062-44e3-a49d-d3e7c452c78c None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.552541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea71616-6062-44e3-a49d-d3e7c452c78c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.552541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea71616-6062-44e3-a49d-d3e7c452c78c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.573065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-eea71616-6062-44e3-a49d-d3e7c452c78c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:26.574354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-eea71616-6062-44e3-a49d-d3e7c452c78c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:26.586964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.594843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.595721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.595999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.596267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.596924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:26.601694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.601694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.609748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 56/116] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:26.610799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.653923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 61/117] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:26.655584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.656055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.656538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.675148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:26.676718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:26.721106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.727873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38e08b9a-008f-48e0-ac55-a75787702989 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.751036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 62/118] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:26.752598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.785093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.785518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.785807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.786084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.875481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:26.882170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0d6a7fd-e0c5-447a-b84b-5801e5162b37 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:26.883101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.885508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.960285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:26.961606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:26.962076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:26.962537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:26.964326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:26.966392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:26.968815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 63/119] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:26.970060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:26.990437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 57/120] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 380 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:26.990902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.009939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.101715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.104372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f23b380f-88a8-4e6c-952c-f5980bde2863 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.106846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d687c9b-f480-434f-97ab-efa9929640cd None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.139891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.141080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.141583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.142194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.143865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.144500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.144775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.145043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.146151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.147714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.149188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 64/121] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:27.150029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.155617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:27.160587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:27.161444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:27.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 58/122] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:26 2022] GET /identity/v3/roles/member => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:27.162862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.266414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.270630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93732b-3e5a-4e66-8620-14785dfc25bd None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.272150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d8e4c04-d24d-4367-a88e-915dc6f732cc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.309896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.310532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.310841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.311198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.312263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.313625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.313774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.314705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.315224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.320690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 65/123] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:27.320821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.320919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.321460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.322979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.326016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 59/124] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:27.328964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.359295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.428028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389f9a10-fddc-4397-8e78-20fc40e94071 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.437791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e43caf1a-88ed-4da1-b183-74c7a2da26d0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.464975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.465608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.465893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.466171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.471941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.472567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.472846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.473114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.474217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.475684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.476394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:27.477536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 60/125] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:27.478140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:27.486320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:27.487271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 66/126] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/roles/member => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:27.492830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.521733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.578284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.591702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76eb938d-c28f-42e6-b6af-eacc58f4d2aa None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.598225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7fdd20-45d3-49a9-bfc2-cb1f59585000 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.629102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.629723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.630007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.630292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.630980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:27.631288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.632670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.635550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.636378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.637432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.637978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/e6f8c1e119d74e408654eed7e007c7f0/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.640309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] RBAC: Authorizing `identity:create_grant(project_id=1d0285a452bf4bb285cfc1f65cacd3ac, user_id=e6f8c1e119d74e408654eed7e007c7f0, role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.641628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 61/127] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:27.641628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.646703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.742632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31f509df-80bf-4e11-8968-e592146283b8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:27.757657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 67/128] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:27 2022] PUT /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/e6f8c1e119d74e408654eed7e007c7f0/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:27.758376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.797642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:27.798382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:27.799242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:27.799528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:27.811818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] RBAC: Adding query filter params (role.id=29d6af9cebaf4223864fe1061ee89e17, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=4d1affdb59974e16940870b99847401e) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:27.811818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:27.811818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:27.812222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 62/129] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/role_assignments?user.id=4d1affdb59974e16940870b99847401e&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=29d6af9cebaf4223864fe1061ee89e17 => generated 259 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:27.813310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:27.853971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.932302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:27.967999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61756f5c-ff04-43ff-b9f4-652832508beb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.013119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.013935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.014659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.015889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e76cc06f-7c72-4bed-919d-916d0fab128d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.018349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/3c05ec42fc16441e83ec77e10313ea85/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.020269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d0285a452bf4bb285cfc1f65cacd3ac, user_id=3c05ec42fc16441e83ec77e10313ea85, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.021711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.053446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.054064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.054345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.054619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.055337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:28.055646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.057039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.067322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 63/130] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:27 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:28.069691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:28.143026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 68/131] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:27 2022] PUT /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/3c05ec42fc16441e83ec77e10313ea85/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:28.147212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:28.168953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:28.216927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8984567b-455b-4b1e-b389-d9c9ba735ef0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.270134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.270838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.271145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.271426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.272698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.277416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.277416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 64/132] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:28 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:28.282897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:28.402899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93d738c7-0b1c-42dc-bb4d-c476d4dca522 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.432142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7328414-0d83-458c-bcce-5820aefd3bf5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.494431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.495213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.495392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.495528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.495831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.496054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.496380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.496660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.497494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.497879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.499351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.499455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.501105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 69/133] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:28 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 357 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:28.501433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 65/134] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:28 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:28.510804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:28.511658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:28.526852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:28.538480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:28.630247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ed42543-a741-4e94-b93c-b3fd781a5647 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.669200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.669902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.670187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.670460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] PATH_INFO: `/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.672463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] RBAC: Authorizing `identity:get_project(project_id=b23d49abd36d42b1a2f35c94fd0590f3)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.674065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.676110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 70/135] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:28 2022] GET /identity/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3 => generated 312 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:28.683467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:28.695464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa38c47f-1f74-453e-bfa7-68988895b50e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.760514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28121266-91c3-44e3-b07e-4440fcef5f7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.761207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28121266-91c3-44e3-b07e-4440fcef5f7a None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.761546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28121266-91c3-44e3-b07e-4440fcef5f7a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.761933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28121266-91c3-44e3-b07e-4440fcef5f7a None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/e6f8c1e119d74e408654eed7e007c7f0/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.764322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28121266-91c3-44e3-b07e-4440fcef5f7a None admin] RBAC: Authorizing `identity:create_grant(project_id=1d0285a452bf4bb285cfc1f65cacd3ac, user_id=e6f8c1e119d74e408654eed7e007c7f0, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.765784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28121266-91c3-44e3-b07e-4440fcef5f7a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.779787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:28.798973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705c40ff-b0d6-4984-b0b8-ea741dc187a3 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:28.834317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:28.835010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.835349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.835625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] PATH_INFO: `/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3/users/e6f8c1e119d74e408654eed7e007c7f0/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.837690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] RBAC: Authorizing `identity:create_grant(project_id=b23d49abd36d42b1a2f35c94fd0590f3, user_id=e6f8c1e119d74e408654eed7e007c7f0, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:28.839153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:28.896370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 66/136] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:28 2022] PUT /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/e6f8c1e119d74e408654eed7e007c7f0/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 385 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:16:28.897850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4622a299-99ed-4c72-aabe-c37a5e79722a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.898145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4622a299-99ed-4c72-aabe-c37a5e79722a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.898423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4622a299-99ed-4c72-aabe-c37a5e79722a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.899919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 67/137] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:28.906343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79147291-1597-4731-a9ac-b3e21792336b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:28.906678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79147291-1597-4731-a9ac-b3e21792336b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:28.906952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79147291-1597-4731-a9ac-b3e21792336b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:28.925239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 71/138] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:28 2022] PUT /identity/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3/users/e6f8c1e119d74e408654eed7e007c7f0/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 242 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:16:28.928829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-79147291-1597-4731-a9ac-b3e21792336b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:28.930192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-79147291-1597-4731-a9ac-b3e21792336b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:29.032531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 68/139] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:29.430986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:29.559959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a45c162-1fe2-4359-90f7-23fe919ed48a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:29.609127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a20aacb-8f0b-4f84-9a29-ea9bf07e8caa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:29.609765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a20aacb-8f0b-4f84-9a29-ea9bf07e8caa None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:29.610038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a20aacb-8f0b-4f84-9a29-ea9bf07e8caa None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:29.610305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a20aacb-8f0b-4f84-9a29-ea9bf07e8caa None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:29.615458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a20aacb-8f0b-4f84-9a29-ea9bf07e8caa None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:29.616912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a20aacb-8f0b-4f84-9a29-ea9bf07e8caa None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:29.618497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 72/140] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:29.624861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79147291-1597-4731-a9ac-b3e21792336b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:29.627028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:29.827185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79147291-1597-4731-a9ac-b3e21792336b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:29.883637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-428213e2-b6ba-4bed-b5f9-1c94a29f0b9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:29.884190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428213e2-b6ba-4bed-b5f9-1c94a29f0b9d None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:29.887012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428213e2-b6ba-4bed-b5f9-1c94a29f0b9d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:29.887012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-428213e2-b6ba-4bed-b5f9-1c94a29f0b9d None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:29.887012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428213e2-b6ba-4bed-b5f9-1c94a29f0b9d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:29.887787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-428213e2-b6ba-4bed-b5f9-1c94a29f0b9d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:30.015962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 69/141] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:29 2022] POST /identity/v3/users => generated 283 bytes in 392 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:30.748563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e28586-9a9c-4e09-9784-29a97c5ad2c7 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:30.749586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e28586-9a9c-4e09-9784-29a97c5ad2c7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:30.750063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13e28586-9a9c-4e09-9784-29a97c5ad2c7 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:30.751642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 73/142] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:30.757381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe602af-ee53-41b3-8b1e-7f379a884b4e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:30.757815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe602af-ee53-41b3-8b1e-7f379a884b4e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:30.757992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe602af-ee53-41b3-8b1e-7f379a884b4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:30.781995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5fe602af-ee53-41b3-8b1e-7f379a884b4e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:30.783253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5fe602af-ee53-41b3-8b1e-7f379a884b4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:30.850485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:30.868711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 70/143] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:31.080447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be96dd2f-2539-45be-8298-2aece87beb8e None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.081047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be96dd2f-2539-45be-8298-2aece87beb8e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.081146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be96dd2f-2539-45be-8298-2aece87beb8e None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.082617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 74/144] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:31.088033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.088320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.088600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.113552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:31.114841 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:31.196209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 71/145] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:31.206465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be96dd2f-2539-45be-8298-2aece87beb8e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:31.219933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:31.325200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be96dd2f-2539-45be-8298-2aece87beb8e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:31.370237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:31.371508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.372004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.372344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.389180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:31.394413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:31.395366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:31.396137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 75/146] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:31 2022] GET /identity/v3/roles/reader => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:31.403708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:31.548145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab38c1a-758c-4d34-85c4-4cd540f58002 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:31.573328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:31.598284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:31.598284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.598284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.598284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.598639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:31.598639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:31.599885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7167d569-6194-46fd-af4f-919e85f3cf98 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:31.614666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 72/147] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:31 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:31.620229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:31.631971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab284ad7-dc71-4536-a912-e3e4ac5fe416 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.632272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab284ad7-dc71-4536-a912-e3e4ac5fe416 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.632516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab284ad7-dc71-4536-a912-e3e4ac5fe416 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.634103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 73/148] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:31.640892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.641171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.641446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.671569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:31.672818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:31.722638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:31.763265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1e8ddc-9e80-49fd-9b8a-fed23f54bc8b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:31.809130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 74/149] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:31.811156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:31.836365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:31.837004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:31.837280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:31.837545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] PATH_INFO: `/v3/users/4d1affdb59974e16940870b99847401e` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:31.840835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] RBAC: Authorizing `identity:get_user(user_id=4d1affdb59974e16940870b99847401e)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:31.842311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:31.844004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 76/150] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:31 2022] GET /identity/v3/users/4d1affdb59974e16940870b99847401e => generated 289 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:31.849639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:32.007151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c652e7b-8b49-4492-9443-8a0c5e3519f4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.026390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6abeb0ba-5e26-45c0-8af7-d831fd2da1cb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.042208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:32.109164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.109725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.110008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.110282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.118027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.118670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.118983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.119259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.123352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:32.124820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:32.136752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 77/151] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:31 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:32.137830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:32.141243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:32.157552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:32.158437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:32.159805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 75/152] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 349 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:32.160469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:32.164549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:32.356719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b0ac8cd-1b50-4a96-82a5-be32ee758c22 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.400051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.401155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.402625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.403095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.404297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:32.404913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:32.406754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:32.420057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 76/153] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:32 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:32.421182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:32.436763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1c6f7d-afe5-48d3-9c34-0d642f10beb2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.474815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:32.518875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.519551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.519892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.520177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/4d1affdb59974e16940870b99847401e/roles/29d6af9cebaf4223864fe1061ee89e17` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.522992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d0285a452bf4bb285cfc1f65cacd3ac, user_id=4d1affdb59974e16940870b99847401e, role_id=29d6af9cebaf4223864fe1061ee89e17)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:32.524450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:32.575293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f56ead-7916-4c0b-9f2f-da89a62463f3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.602690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 78/154] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:32 2022] PUT /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/users/4d1affdb59974e16940870b99847401e/roles/29d6af9cebaf4223864fe1061ee89e17 => generated 0 bytes in 465 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:32.603637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:32.644182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666aafee-c0b9-43dc-a4f8-d9ae9e688bcf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.644887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666aafee-c0b9-43dc-a4f8-d9ae9e688bcf None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.645164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666aafee-c0b9-43dc-a4f8-d9ae9e688bcf None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.645432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666aafee-c0b9-43dc-a4f8-d9ae9e688bcf None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.668834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666aafee-c0b9-43dc-a4f8-d9ae9e688bcf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-666aafee-c0b9-43dc-a4f8-d9ae9e688bcf None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:32.673966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:32.674992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 77/155] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:32.676618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270541db-88e0-46a5-b7b3-50dc527f11a7 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.677347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270541db-88e0-46a5-b7b3-50dc527f11a7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.677347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-270541db-88e0-46a5-b7b3-50dc527f11a7 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.680186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 78/156] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:32.680186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99cdbf8-1128-4290-a7f7-2eeaaca0bd4d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.680186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99cdbf8-1128-4290-a7f7-2eeaaca0bd4d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.680186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99cdbf8-1128-4290-a7f7-2eeaaca0bd4d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.681194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 79/157] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:32 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:32.681889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99cdbf8-1128-4290-a7f7-2eeaaca0bd4d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:32.711227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:32.746241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8990f3ce-7bdc-48fe-9666-dcf0eb813733 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.785467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7b3a5cc-78ae-4e9c-bd64-eecd80f1659e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.786233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3a5cc-78ae-4e9c-bd64-eecd80f1659e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.786424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3a5cc-78ae-4e9c-bd64-eecd80f1659e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.786704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7b3a5cc-78ae-4e9c-bd64-eecd80f1659e None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.788287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b3a5cc-78ae-4e9c-bd64-eecd80f1659e None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:32.789987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7b3a5cc-78ae-4e9c-bd64-eecd80f1659e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:32.791884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 79/158] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:32 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:32.793111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292b5444-a6a6-4afb-9425-8a5b65a3819d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.793390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292b5444-a6a6-4afb-9425-8a5b65a3819d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.793665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292b5444-a6a6-4afb-9425-8a5b65a3819d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.815501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-292b5444-a6a6-4afb-9425-8a5b65a3819d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:32.816945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-292b5444-a6a6-4afb-9425-8a5b65a3819d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:32.880280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:32.894874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 80/159] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:32.895720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99cdbf8-1128-4290-a7f7-2eeaaca0bd4d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:32.896339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.896626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.896961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.923907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:32.925045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:32.962260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:32.963469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:32.963469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:32.963469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:32.964466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:32.964781 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:32.966185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:32.975735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 80/160] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:32.976813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.064806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 81/161] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:32 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:33.065683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.069602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:33.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b22bfb-4653-41a8-89e5-6b7710dee68b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:33.185624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6408ce2c-7a27-4aca-92a6-16fc5b33d7fd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:33.222519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:33.223194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:33.223470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:33.223743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] PATH_INFO: `/v3/users/3c05ec42fc16441e83ec77e10313ea85` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:33.224890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] RBAC: Authorizing `identity:get_user(user_id=3c05ec42fc16441e83ec77e10313ea85)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:33.226360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:33.228220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 82/162] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:33 2022] GET /identity/v3/users/3c05ec42fc16441e83ec77e10313ea85 => generated 247 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:33.235028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.275502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:33.276134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:33.276412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:33.276681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:33.307249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:33.327050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:33.354801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39f5534f-88be-45eb-a83d-dfd17df1848e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:33.407222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:33.408001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:33.408569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:33.408842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:33.432293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:33.433789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:33.515945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:33.517663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:33.601018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 81/163] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:32 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 625 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:33.601878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.607576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:33.645870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 83/164] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:33 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:16:33.646982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.709618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:33.756169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8464a2f-5935-4456-9982-b9622af975ee None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:33.787213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ee2c16-50b9-4cc9-9468-4fdfafa3ee98 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:33.796375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:33.797598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:33.798268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:33.798696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:33.800228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:33.801869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:33.804013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 84/165] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:33 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:33.805204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.838388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:33.852780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:33.853431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:33.858355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:33.858499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:33.879152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:33.885724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:33.887641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:33.888645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:33.890653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 82/166] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:33.891507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:33.998612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421e512a-1ef9-4ce4-8efc-9fb79781ba4d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.049367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.050363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.050937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.051632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.054334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:34.056099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:34.058121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 85/167] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:33 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:34.059183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:34.110842 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60ef64ea-f82b-4c8a-8401-2be14af319ce None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.139427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:34.172066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.172721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.173000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.173270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.196122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:34.201674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:34.202605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:34.203508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:34.211074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 83/168] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 312 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:34.211833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:34.283133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:34.289120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e29dbc-7500-43ec-800d-c63acb4b5530 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.367500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.368120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.368401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.368672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.369411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] RBAC: Adding query filter params (role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=e6f8c1e119d74e408654eed7e007c7f0) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:34.369717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:34.371715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:34.377677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 86/169] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/role_assignments?user.id=e6f8c1e119d74e408654eed7e007c7f0&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 608 bytes in 319 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:34.378570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:34.441864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d9dbd85-71f3-4568-b02f-fe5bb0b84d6d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.520050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.520662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.520942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.521216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.521889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:34.522195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:34.523647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:34.546287 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 84/170] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 335 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:34.548520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:34.592670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13bef17c-be84-481f-8b6b-5415087cc366 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.623931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:34.688516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.689138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.689414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.689889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.690638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=3c05ec42fc16441e83ec77e10313ea85) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:34.690966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:34.692439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:34.709319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 87/171] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/role_assignments?user.id=3c05ec42fc16441e83ec77e10313ea85&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 331 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:34.710187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:34.753756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:34.812803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-916d776c-e8d1-4ef1-b895-d00fe391e6c1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.869773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2358acf-c6e7-47fb-bedc-82a9a244cc39 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:34.913258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.913817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.914094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.914365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.919592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:34.920603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:34.921092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:34.921320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:34.921495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:34.921771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:34.922443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:34.922722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 88/172] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:34.923118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:34.924531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:34.933265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:34.942256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 85/173] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 395 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:34.949608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:35.074623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc66f408-5b79-4dec-99b7-3fcf208fda02 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:35.082374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:35.122628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3b35d4-d4b9-4ff2-9904-50fb1ce9a176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:35.123271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b35d4-d4b9-4ff2-9904-50fb1ce9a176 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.123553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b35d4-d4b9-4ff2-9904-50fb1ce9a176 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.123822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3b35d4-d4b9-4ff2-9904-50fb1ce9a176 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.209508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b35d4-d4b9-4ff2-9904-50fb1ce9a176 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:35.211135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3b35d4-d4b9-4ff2-9904-50fb1ce9a176 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:35.264322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:35.281012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-807444a2-0313-4cd7-9a05-8e6e4e4faaa2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:35.321876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 89/174] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:16:35.323439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ad7192-c66a-49cd-97ac-8ab9b83fb75a None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.324383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ad7192-c66a-49cd-97ac-8ab9b83fb75a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.324660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8ad7192-c66a-49cd-97ac-8ab9b83fb75a None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.326533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 90/175] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:35.327018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:35.332248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.332530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.332881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.353481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:35.355927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:35.395510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:35.396154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.396434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.396705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] PATH_INFO: `/v3/users/e6f8c1e119d74e408654eed7e007c7f0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.397886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] RBAC: Authorizing `identity:get_user(user_id=e6f8c1e119d74e408654eed7e007c7f0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:35.399367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:35.402106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 86/176] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:34 2022] GET /identity/v3/users/e6f8c1e119d74e408654eed7e007c7f0 => generated 275 bytes in 453 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:35.403033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:35.434203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 91/177] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:35.434936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:35.585519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eed637d-47bd-40eb-96c9-66455a2dd053 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:35.614407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:35.614712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2607b71a-6a6b-4b70-a33b-a6d08b0ee712 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:35.627084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962e380a-f461-4e84-a99d-5d7c975b7ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:35.638369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e380a-f461-4e84-a99d-5d7c975b7ca5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.638369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e380a-f461-4e84-a99d-5d7c975b7ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.638369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962e380a-f461-4e84-a99d-5d7c975b7ca5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.659898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ac7230-4265-4f92-b29b-12329a9f3055 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:35.661198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac7230-4265-4f92-b29b-12329a9f3055 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.661983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac7230-4265-4f92-b29b-12329a9f3055 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.662526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ac7230-4265-4f92-b29b-12329a9f3055 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.664370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ac7230-4265-4f92-b29b-12329a9f3055 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:35.666306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ac7230-4265-4f92-b29b-12329a9f3055 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:35.669320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 87/178] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:35 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 267 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:35.670794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a215e36-648d-4e64-915b-03e1e6364ba5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.671284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a215e36-648d-4e64-915b-03e1e6364ba5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.671755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a215e36-648d-4e64-915b-03e1e6364ba5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.673332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 88/179] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:35.674558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561dd153-6fc6-4e84-b4cb-5fe3fa8663be None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.675235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561dd153-6fc6-4e84-b4cb-5fe3fa8663be None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.676588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561dd153-6fc6-4e84-b4cb-5fe3fa8663be None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.678039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 89/180] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:35.690098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561dd153-6fc6-4e84-b4cb-5fe3fa8663be None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:35.722594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e380a-f461-4e84-a99d-5d7c975b7ca5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:35.724122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962e380a-f461-4e84-a99d-5d7c975b7ca5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:35.744816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:35.828596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 92/181] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:35.830055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81945556-db5e-4028-98e8-8bee08de339f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.830592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81945556-db5e-4028-98e8-8bee08de339f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.830900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81945556-db5e-4028-98e8-8bee08de339f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.832421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561dd153-6fc6-4e84-b4cb-5fe3fa8663be None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:35.848986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-81945556-db5e-4028-98e8-8bee08de339f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:35.850095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-81945556-db5e-4028-98e8-8bee08de339f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:35.867868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:35.868266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:35.868876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.869156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.869427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.870142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=e6f8c1e119d74e408654eed7e007c7f0) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:35.871003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:35.872530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca829f2d-e5a5-4922-aeaf-fedc3ebf01b6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:35.878358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 90/182] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:35 2022] GET /identity/v3/role_assignments?user.id=e6f8c1e119d74e408654eed7e007c7f0&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=1448af55b1904e1199e3954829e9815a => generated 608 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:35.879691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:35.879976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:35.880245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:35.903698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:35.904938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:35.936221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 93/183] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:35.937232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81945556-db5e-4028-98e8-8bee08de339f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:35.994158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 91/184] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:35.996405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.050799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.129550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81945556-db5e-4028-98e8-8bee08de339f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.171681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b03cd2a8-cc3b-4268-beaa-16f16ff69f9a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.184340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-259f61e3-7c0b-46f3-9904-387f281dece8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.184980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259f61e3-7c0b-46f3-9904-387f281dece8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.185260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259f61e3-7c0b-46f3-9904-387f281dece8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.185528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-259f61e3-7c0b-46f3-9904-387f281dece8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.202204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-259f61e3-7c0b-46f3-9904-387f281dece8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-259f61e3-7c0b-46f3-9904-387f281dece8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:36.208486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:36.209386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:36.210164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:36.212598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 94/185] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:36.215286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02794152-0048-4e5f-b62a-4ddefcbb4c89 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.215870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02794152-0048-4e5f-b62a-4ddefcbb4c89 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.216334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02794152-0048-4e5f-b62a-4ddefcbb4c89 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.218873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 95/186] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:36 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:36.219810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02794152-0048-4e5f-b62a-4ddefcbb4c89 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.222610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0157203-c998-4c75-b196-f679a5532631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.223473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0157203-c998-4c75-b196-f679a5532631 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.223665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0157203-c998-4c75-b196-f679a5532631 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.223995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0157203-c998-4c75-b196-f679a5532631 None admin] PATH_INFO: `/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.225797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0157203-c998-4c75-b196-f679a5532631 None admin] RBAC: Authorizing `identity:get_project(project_id=b23d49abd36d42b1a2f35c94fd0590f3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.226950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0157203-c998-4c75-b196-f679a5532631 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.228709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 92/187] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:35 2022] GET /identity/v3/projects/b23d49abd36d42b1a2f35c94fd0590f3 => generated 312 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:36.230402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.230689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.231040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.249820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.260117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:36.264703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:36.337086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 93/188] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:36.338064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.350519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02794152-0048-4e5f-b62a-4ddefcbb4c89 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.369162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.389149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.389690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.389972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.390248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.390971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:36.391278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.392741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.402870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 96/189] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:36.403713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.456314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd17f05c-e928-48e1-aa9d-2622422d3bd2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.487089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.493596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.494254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.494544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.494839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.496085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=b23d49abd36d42b1a2f35c94fd0590f3, user.id=e6f8c1e119d74e408654eed7e007c7f0) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:36.496401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.497864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.510822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 94/190] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/role_assignments?user.id=e6f8c1e119d74e408654eed7e007c7f0&scope.project.id=b23d49abd36d42b1a2f35c94fd0590f3&role.id=1448af55b1904e1199e3954829e9815a => generated 608 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:36.510822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.520554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1190bec6-26a5-45c4-a5ed-f1372231ff16 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.567454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.568101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.568379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.569138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.585987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:36.594312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:36.595283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:36.596307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 97/191] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:36.596961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.617744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.635074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b518376c-108e-49df-aca0-31e0f2883423 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.672833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.673393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.673674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.673946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.675164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.676637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.678421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 95/192] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:36.679246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.726185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-379165d2-c4b0-4abb-98c7-0a3f1d1cef02 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.769842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.770492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.770759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.771482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] PATH_INFO: `/v3/users/3e129736df4c49dfaf32196faf9599fa` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.774443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.787314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] RBAC: Authorizing `identity:get_user(user_id=3e129736df4c49dfaf32196faf9599fa)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.788773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.794652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 98/193] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/users/3e129736df4c49dfaf32196faf9599fa => generated 283 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:36.794652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.794950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cff7de4-49e7-469f-b128-240dc5213555 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.829039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.829640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.829913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.830179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.831115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:36.831413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.832772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.840969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 96/194] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:36.843624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.865974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:36.890524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f11f54d0-b076-492a-9260-3bcc152a619c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.923220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.923784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.924065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.924339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.934542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:36.939457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:36.940853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 99/195] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/roles/reader => generated 84 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:36.941780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:36.949222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6672727-e853-407a-8926-e9f06fac7653 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:36.984876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:36.985484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:36.985763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:36.986036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:36.986680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:36.988139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:36.998481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.040661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6405157d-f444-4b77-a8d7-71f05a3e1e75 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.072420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.073017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.073297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.074089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.102811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 97/196] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:36 2022] POST /identity/v3/users => generated 297 bytes in 261 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:37.104154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.155818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.157317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.215805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.218529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.240370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89dbcff5-f671-4a1d-b868-7646a856663e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.274700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 100/197] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 329 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:37.274700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.283515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.284427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.284705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.285689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] PATH_INFO: `/v3/users/3e129736df4c49dfaf32196faf9599fa` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.286121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] RBAC: Authorizing `identity:get_user(user_id=3e129736df4c49dfaf32196faf9599fa)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.288439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.290086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 98/198] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/users/3e129736df4c49dfaf32196faf9599fa => generated 283 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:37.290086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.410682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d93926-ac9d-40b0-9f51-c6fbf28c17f7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.427725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3bc82f2-f75f-4631-836a-28d94b0efe50 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.460606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.461218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.461491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.461759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.462420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] RBAC: Adding query filter params (name=reader) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:37.462799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.464693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.472901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.473480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.473759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.473990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 101/199] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:37.474375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.474833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.475722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.477197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.478919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 99/200] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:37.483026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.530000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.535383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.626974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09aea24d-992a-449e-ba80-46ba59138923 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.627846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c4914a3-5122-4389-8043-f0e0e601b55c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.669248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.669462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.670113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.670384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.670575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.670993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.671114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.671508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] PATH_INFO: `/v3/users/4d1affdb59974e16940870b99847401e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.672790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] RBAC: Authorizing `identity:get_user(user_id=4d1affdb59974e16940870b99847401e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.674542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.676436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 100/201] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/users/4d1affdb59974e16940870b99847401e => generated 289 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:37.681693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.752400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.754163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.769509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.789462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f579bbba-8f9a-4140-afd5-ca8f40e12c20 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:37.807236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:37.825929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:37.826543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:37.826856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:37.827127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:37.827847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=3e129736df4c49dfaf32196faf9599fa) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:37.828155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:37.829658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:37.835538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 101/202] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/role_assignments?user.id=3e129736df4c49dfaf32196faf9599fa&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:37.843872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.867353 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 102/203] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:37.873801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:37.989388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ce4c102-17d6-4e82-9669-2375a6d0c2a0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:38.029711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:38.030385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:38.030799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:38.031079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:38.032242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:38.032724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:38.037705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:38.039571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 103/204] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:38.047396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:38.112239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e052610e-aad6-4cd5-bae7-5c2b456717cd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:38.160905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:38.161917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:38.162199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:38.162474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:38.195529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:38.248250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ea3c957-1567-4dde-8fc5-75aa470125ac None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:38.285442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:38.285977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:38.286318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:38.286599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:38.287356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] RBAC: Adding query filter params (role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=3e129736df4c49dfaf32196faf9599fa) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:38.287665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:38.289745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:38.295761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:38.297470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:38.303310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 104/205] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:38 2022] GET /identity/v3/role_assignments?user.id=3e129736df4c49dfaf32196faf9599fa&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 259 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:38.454418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 102/206] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 618 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:38.460694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:38.544950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:38.601409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2e21087-bc0c-4162-8918-bb656453d1a6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:38.638643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0860b863-41a3-43c4-94ad-1f89f0702d3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:38.639350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860b863-41a3-43c4-94ad-1f89f0702d3c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:38.639631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860b863-41a3-43c4-94ad-1f89f0702d3c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:38.639897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0860b863-41a3-43c4-94ad-1f89f0702d3c None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:38.641013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0860b863-41a3-43c4-94ad-1f89f0702d3c None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:38.642449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0860b863-41a3-43c4-94ad-1f89f0702d3c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:38.644437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 105/207] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:38 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:38.650167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:38.713719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:38.795798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d912c636-60d9-4f71-8498-a3673df0f7cc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:38.842630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:38.843268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:38.843543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:38.843810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:38.844528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] RBAC: Adding query filter params (role.id=29d6af9cebaf4223864fe1061ee89e17, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac, user.id=4d1affdb59974e16940870b99847401e) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:38.844829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:38.846273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8831d6f2-baa7-4a5d-94d8-a0bf11ca3692 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:38.854452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 103/208] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:38 2022] GET /identity/v3/role_assignments?user.id=4d1affdb59974e16940870b99847401e&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=29d6af9cebaf4223864fe1061ee89e17 => generated 608 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:39.590968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1e93ea-ebc5-4bf8-9500-5a65f95f08e1 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:39.591977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1e93ea-ebc5-4bf8-9500-5a65f95f08e1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:39.592551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1e93ea-ebc5-4bf8-9500-5a65f95f08e1 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:39.594349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 106/209] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:39.600048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:39.600847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:39.601318 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:39.620689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:39.622200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:39.678043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b3373f-d34e-40ad-bfd9-de8a160dd9e9 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:39.678665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b3373f-d34e-40ad-bfd9-de8a160dd9e9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:39.679165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1b3373f-d34e-40ad-bfd9-de8a160dd9e9 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:39.681247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 107/210] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:39.688133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:39.688646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:39.689117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:39.695435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 104/211] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:39.711374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:39.712571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:39.781992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 108/212] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:39.970043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.009745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:40.054977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.076072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19ebf78b-03fe-4b92-9d18-4053f6e6f856 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:40.105218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:40.121806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:40.123079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.123426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.123666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.137736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:40.144230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:40.145105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:40.145905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:40.146497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 105/213] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:39 2022] GET /identity/v3/roles/member => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:40.152932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.165161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f18469e-de67-46b5-abb5-1d295166a863 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:40.206734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:40.207616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.207872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.208140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.209249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:40.210832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:40.217484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 109/214] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:40.219283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.279260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:40.284607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82db7745-5e73-4af8-a2d8-965174894231 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:40.330520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:40.331160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.331459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.331734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.332395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:40.332695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:40.334074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:40.343689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 106/215] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:40 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:40.344165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b1d031-dcc8-4a01-9269-120eda82e022 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:40.349785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.364451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:40.391906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1013b103-3166-4be9-b4bb-94b4c917d26e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:40.393002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013b103-3166-4be9-b4bb-94b4c917d26e None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.393480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013b103-3166-4be9-b4bb-94b4c917d26e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.395132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1013b103-3166-4be9-b4bb-94b4c917d26e None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.396048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013b103-3166-4be9-b4bb-94b4c917d26e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:40.397735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1013b103-3166-4be9-b4bb-94b4c917d26e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:40.463294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-badd1ad8-31ab-4547-bd8b-d64d558051ce None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:40.498941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:40.499676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.499967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.500221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] PATH_INFO: `/v3/users/25063445b44f41adac6f0ddb96dc9f57` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.504583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 110/216] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:40 2022] POST /identity/v3/users => generated 297 bytes in 286 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:16:40.508690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ed152a-23a8-434a-bf77-e984179a425c None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.509047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ed152a-23a8-434a-bf77-e984179a425c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.509393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ed152a-23a8-434a-bf77-e984179a425c None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.511721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 111/217] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:40.519400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e3dd0-5003-4774-bb7e-495319cb4097 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.520047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e3dd0-5003-4774-bb7e-495319cb4097 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.520692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948e3dd0-5003-4774-bb7e-495319cb4097 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.523022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] RBAC: Authorizing `identity:get_user(user_id=25063445b44f41adac6f0ddb96dc9f57)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:40.524861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:40.531953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 107/218] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:40 2022] GET /identity/v3/users/25063445b44f41adac6f0ddb96dc9f57 => generated 297 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:40.533203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.544842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-948e3dd0-5003-4774-bb7e-495319cb4097 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:40.546411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-948e3dd0-5003-4774-bb7e-495319cb4097 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:40.555615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:40.647483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 112/219] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:40.666474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f828892-48cb-45fa-9555-07b182a7fef0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:40.709350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:40.709971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.710249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.710517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.781496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:40.783104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:40.787989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04189fde-8139-4234-bfd2-739b2dc5f51c None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.787989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04189fde-8139-4234-bfd2-739b2dc5f51c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.787989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04189fde-8139-4234-bfd2-739b2dc5f51c None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.789379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 113/220] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:40.795464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:40.795742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:40.796013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:40.802916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:40.826825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:40.828506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:40.904889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 108/221] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:40 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:40.911307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.937821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 114/222] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:40.939029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:40.974890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.049566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52ad677b-51cd-4bde-89da-183dfde8dfdc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.068484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.075399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b49f02d-1f2c-4809-adb1-c9ef9c6f21ff None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.086701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.087600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.088075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.088480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.090820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.092656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.099117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 109/223] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:40 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:41.101731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.116602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.117245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.117526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.117799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.130092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:41.135481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:41.137143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:41.137938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 115/224] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:40 2022] GET /identity/v3/roles/admin => generated 83 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:41.147930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.297328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cb76da5-4067-4f80-aea0-5f7133df7004 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.300136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-697b8081-00d4-4470-b3fb-5914224ed7d4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.303065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.345619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.346243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.346520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.346807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.347866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=25063445b44f41adac6f0ddb96dc9f57) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:41.348173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.349616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.351009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.351615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.351886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.352151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.352819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:41.353117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.354499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.359092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 110/225] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/role_assignments?user.id=25063445b44f41adac6f0ddb96dc9f57&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=1448af55b1904e1199e3954829e9815a => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:41.360202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.364763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 116/226] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:41.372025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.462460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.562987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e6d408a-0869-44dc-bee6-e7b75738cf06 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.566304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-331dc665-99c3-48b2-9785-0af56e19bf4d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.605531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.606249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.606524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.606816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] PATH_INFO: `/v3/users/3e129736df4c49dfaf32196faf9599fa` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.607972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] RBAC: Authorizing `identity:get_user(user_id=3e129736df4c49dfaf32196faf9599fa)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.609414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.610114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.610764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.611069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.611346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.611466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 117/227] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/users/3e129736df4c49dfaf32196faf9599fa => generated 283 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:41.616823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.641791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:41.647168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:41.648023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:41.648805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 111/228] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 289 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:41.654593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.671206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.744656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86b57e04-0ef5-4080-888f-ee5fa0063779 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.780183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.781156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.781627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.782109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.783479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.785142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.787005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 118/229] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:41.798127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:41.904605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6282d05-3336-4e5b-8a7f-da5098859216 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.921468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.921633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:41.933645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07b4dde-13ef-4ee1-bce6-ea6252c33855 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:41.968859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.969890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.970453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.970950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/3e129736df4c49dfaf32196faf9599fa/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.971475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:41.972101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:41.972376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:41.972645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:41.973312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:41.973616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.974290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c31ce5c991046b0a793e0e05d08cf32, user_id=3e129736df4c49dfaf32196faf9599fa, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:41.975041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.985741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:41.989968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 112/230] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 336 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:41.995797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:42.081564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 119/231] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:41 2022] PUT /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/3e129736df4c49dfaf32196faf9599fa/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 284 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:42.146453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db4f4b8a-079f-46b6-b849-9e8f37097c8d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:42.192233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:42.192844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:42.193123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:42.193428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] PATH_INFO: `/v3/users/3e129736df4c49dfaf32196faf9599fa` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:42.194571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] RBAC: Authorizing `identity:get_user(user_id=3e129736df4c49dfaf32196faf9599fa)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:42.196041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:42.197617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 113/232] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:41 2022] GET /identity/v3/users/3e129736df4c49dfaf32196faf9599fa => generated 283 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:42.204373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:42.328707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aeb1eeb-d070-4633-b2c1-a727698db313 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:42.368469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-884ce3fc-0ebb-4eda-b154-7e5c04b1f289 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:42.369110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884ce3fc-0ebb-4eda-b154-7e5c04b1f289 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:42.369389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884ce3fc-0ebb-4eda-b154-7e5c04b1f289 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:42.369659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-884ce3fc-0ebb-4eda-b154-7e5c04b1f289 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:42.371170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884ce3fc-0ebb-4eda-b154-7e5c04b1f289 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:42.372651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-884ce3fc-0ebb-4eda-b154-7e5c04b1f289 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:42.374320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 120/233] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:42 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:42.380201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:42.425961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:42.486690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-014fdee7-a916-4db6-b9d6-442cf2271ae6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:42.525266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7581ce77-3faa-4f53-81be-61b9973dbe36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:42.525989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581ce77-3faa-4f53-81be-61b9973dbe36 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:42.526296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581ce77-3faa-4f53-81be-61b9973dbe36 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:42.526643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7581ce77-3faa-4f53-81be-61b9973dbe36 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/3e129736df4c49dfaf32196faf9599fa/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:42.528905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7581ce77-3faa-4f53-81be-61b9973dbe36 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c31ce5c991046b0a793e0e05d08cf32, user_id=3e129736df4c49dfaf32196faf9599fa, role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:42.530344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7581ce77-3faa-4f53-81be-61b9973dbe36 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:42.626704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 114/234] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:42 2022] PUT /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/3e129736df4c49dfaf32196faf9599fa/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:43.328869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687478b-a9ee-4a00-bec4-732af1bb6c85 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:43.329396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687478b-a9ee-4a00-bec4-732af1bb6c85 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:43.329396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d687478b-a9ee-4a00-bec4-732af1bb6c85 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:43.331371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 121/235] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:43.336904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:43.337338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:43.337428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:43.350703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a96ea-751a-4b0f-93be-f9093eb33495 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:43.351016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a96ea-751a-4b0f-93be-f9093eb33495 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:43.351524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5a96ea-751a-4b0f-93be-f9093eb33495 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:43.352877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 122/236] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:43.364269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:43.365677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:43.370967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:43.371289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:43.374709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:43.400152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:43.401499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:43.417161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:43.481862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 115/237] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:43.501859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 123/238] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:43.896500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:43.916606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:43.937430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:44.015279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4461f4e-1e2d-4ff0-83e6-9748272afce2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.046040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fa4a2-1a29-488e-abac-ecff73b5d4d0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.054112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32ac5cf5-1b1b-40e0-a9ca-2e951c0291df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.054744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ac5cf5-1b1b-40e0-a9ca-2e951c0291df None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.055414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ac5cf5-1b1b-40e0-a9ca-2e951c0291df None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.055414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32ac5cf5-1b1b-40e0-a9ca-2e951c0291df None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.071597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32ac5cf5-1b1b-40e0-a9ca-2e951c0291df None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-32ac5cf5-1b1b-40e0-a9ca-2e951c0291df None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:44.080365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:44.081428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:44.082384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:44.085334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 116/239] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:43 2022] GET /identity/v3/roles/reader => generated 84 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:44.086660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49243687-7c1c-4bfc-8449-97e2cd86b473 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.087087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49243687-7c1c-4bfc-8449-97e2cd86b473 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.087505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49243687-7c1c-4bfc-8449-97e2cd86b473 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.090540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 117/240] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:44.091707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49243687-7c1c-4bfc-8449-97e2cd86b473 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:44.100673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ffb48c-9b18-4cee-b7c0-f383ade41fa5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.101454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ffb48c-9b18-4cee-b7c0-f383ade41fa5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.101830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ffb48c-9b18-4cee-b7c0-f383ade41fa5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.102413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ffb48c-9b18-4cee-b7c0-f383ade41fa5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.103390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ffb48c-9b18-4cee-b7c0-f383ade41fa5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:44.105073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ffb48c-9b18-4cee-b7c0-f383ade41fa5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:44.107110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 124/241] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:44.111999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.111999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.111999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.128065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:44.129349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:44.139947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:44.211264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 125/242] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:44.212436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:44.223567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:44.237346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49243687-7c1c-4bfc-8449-97e2cd86b473 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.281565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.282907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.283371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.283969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.284963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:44.285449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:44.287331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:44.299336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 118/243] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:44 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:44.309906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:44.437612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e806aa6e-efaf-44e8-83ed-7957f951d539 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.463538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7070c44b-d3f0-40b9-932e-64581b78c0f7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.508547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.510501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.513080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.513664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] PATH_INFO: `/v3/users/56012eb4ea294d299043849dc08ad010` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.531878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.532489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.532766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.533030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.533677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:44.535353 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:44.543968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] RBAC: Authorizing `identity:get_user(user_id=56012eb4ea294d299043849dc08ad010)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:44.545538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:44.547648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 119/244] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:44 2022] GET /identity/v3/users/56012eb4ea294d299043849dc08ad010 => generated 297 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:44.553156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:44.565990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:44.623893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:44.629602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 126/245] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:44 2022] POST /identity/v3/users => generated 293 bytes in 418 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:44.639507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:44.679638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5fad875-e1e8-4bdf-83b6-02444905a3ac None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.771622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25e945fe-919e-4192-99e4-01c1c08a53fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.772652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e945fe-919e-4192-99e4-01c1c08a53fe None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.773156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e945fe-919e-4192-99e4-01c1c08a53fe None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.773651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25e945fe-919e-4192-99e4-01c1c08a53fe None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.832365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0c2b0ec-ae9d-4186-95db-4c0c5c97a355 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:44.872794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93ca92dd-60cb-411f-b269-a78f2beb61f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:44.873832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca92dd-60cb-411f-b269-a78f2beb61f4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.874354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca92dd-60cb-411f-b269-a78f2beb61f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.874847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93ca92dd-60cb-411f-b269-a78f2beb61f4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.888872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93ca92dd-60cb-411f-b269-a78f2beb61f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-93ca92dd-60cb-411f-b269-a78f2beb61f4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:44.894479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:44.895638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:44.896687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:44.898667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 127/246] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:44 2022] GET /identity/v3/roles/member => generated 84 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:44.900046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85abe333-0c65-47c8-846e-20748736a202 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:44.901902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85abe333-0c65-47c8-846e-20748736a202 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:44.902324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85abe333-0c65-47c8-846e-20748736a202 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:44.904350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 128/247] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:44.906507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85abe333-0c65-47c8-846e-20748736a202 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:44.923799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e945fe-919e-4192-99e4-01c1c08a53fe None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:44.925459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25e945fe-919e-4192-99e4-01c1c08a53fe None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:44.937834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:44.978438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:45.035176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85abe333-0c65-47c8-846e-20748736a202 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.043087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 120/248] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 490 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:45.044908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.045191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.045481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.071918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:45.073533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:45.084514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.085126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.085403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.085672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.086337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:45.086632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:45.088111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:45.097973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 129/249] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:44 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:45.098837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:45.165109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 121/250] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:45.166165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:45.220731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:45.228670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83141ac3-1e1a-45f4-994a-e6c7de27a1b9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.257570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:45.272095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.273049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.273541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.274286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.276389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:45.278043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:45.280268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 130/251] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:45 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:45.289156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:45.297040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d3e17b9-ed58-49bd-a050-910efc395291 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.345361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.346259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.346749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.347260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] PATH_INFO: `/v3/users/25063445b44f41adac6f0ddb96dc9f57` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.349191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] RBAC: Authorizing `identity:get_user(user_id=25063445b44f41adac6f0ddb96dc9f57)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:45.351044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:45.353666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 122/252] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:45 2022] GET /identity/v3/users/25063445b44f41adac6f0ddb96dc9f57 => generated 297 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:45.360294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:45.484541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58b11c40-74c8-40d9-936e-5a50d81a71f9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.514322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a80d79aa-abcf-4879-8b54-231db4c66721 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.524090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:45.535499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.536125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.536406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.536675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.537409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] RBAC: Adding query filter params (role.id=29d6af9cebaf4223864fe1061ee89e17, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=56012eb4ea294d299043849dc08ad010) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:45.537718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:45.539277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031afbc8-5573-4861-9731-e45c7274fe55 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:45.545291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 131/253] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:45 2022] GET /identity/v3/role_assignments?user.id=56012eb4ea294d299043849dc08ad010&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=29d6af9cebaf4223864fe1061ee89e17 => generated 259 bytes in 257 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:45.546489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a34b1-86cd-4f48-b2e4-c90609fc9a3a None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.546768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a34b1-86cd-4f48-b2e4-c90609fc9a3a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.555271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3a34b1-86cd-4f48-b2e4-c90609fc9a3a None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.558228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 132/254] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:45 2022] GET /identity => generated 270 bytes in 11 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:45.561282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71bdbe06-bbbf-4e3d-8223-ac875ac3c585 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.561935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdbe06-bbbf-4e3d-8223-ac875ac3c585 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.562210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdbe06-bbbf-4e3d-8223-ac875ac3c585 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.562478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71bdbe06-bbbf-4e3d-8223-ac875ac3c585 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.567672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bdbe06-bbbf-4e3d-8223-ac875ac3c585 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:45.569149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71bdbe06-bbbf-4e3d-8223-ac875ac3c585 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:45.571220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 123/255] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:45 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:45.577002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a34b1-86cd-4f48-b2e4-c90609fc9a3a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:45.577947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.578226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.578499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.613327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:45.614547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:45.707073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:45.769335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 124/256] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 363 bytes (3 switches on core 0) Sep 27 16:16:45.770303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:45.796241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3a34b1-86cd-4f48-b2e4-c90609fc9a3a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.856124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.856799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.857078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.857352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/25063445b44f41adac6f0ddb96dc9f57/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.864110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c31ce5c991046b0a793e0e05d08cf32, user_id=25063445b44f41adac6f0ddb96dc9f57, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:45.865579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:45.935125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1109861d-2f46-4e2b-87cc-9dcd66d3f6be None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:45.943828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:45.944923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 133/257] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:45 2022] PUT /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/25063445b44f41adac6f0ddb96dc9f57/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:45.974709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:45.975383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:45.975656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:45.975924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:45.991387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:45.997948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:45.998937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:46.000595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 125/258] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:16:45 2022] GET /identity/v3/roles/admin => generated 83 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:46.008124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.146651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e896ea-417e-4aa1-be91-366a59d0c527 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:46.163242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:46.192532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:46.193162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:46.193438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:46.193708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:46.194388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:46.194688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:46.196107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:46.208883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 134/259] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:46.214744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.244554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.345774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e214dfbd-edef-4730-b8ce-5b5fc277e37d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:46.377557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:46.378196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:46.378470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:46.378740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] PATH_INFO: `/v3/users/3e129736df4c49dfaf32196faf9599fa` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:46.379892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] RBAC: Authorizing `identity:get_user(user_id=3e129736df4c49dfaf32196faf9599fa)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:46.381333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:46.383264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 126/260] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/users/3e129736df4c49dfaf32196faf9599fa => generated 283 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:46.389607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.432778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:46.435909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01824ef4-0b3b-479c-be8d-832ba8a04fb3 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:46.480073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:46.480842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:46.481117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:46.481889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:46.507903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a14ab88-ad81-4607-9530-dd82e0ad72c8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:46.515379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:46.520369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:46.521244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:46.522048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 135/261] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:46.532379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.563174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:46.564578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:46.565085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:46.565664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:46.573870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:46.647430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cabe52-6306-465a-a044-ba9e67ce3bbc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:46.686954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:46.687481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:46.687757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:46.688023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:46.689056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:46.689371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:46.691020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:46.701163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 136/262] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:46.706963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.722276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:46.724224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:46.788182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:46.817694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:46.832746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6a7bb3e-4f0c-43bb-8ec9-edb899b22c90 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:46.915680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:46.917209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:46.918313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:46.919050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] PATH_INFO: `/v3/users/3e129736df4c49dfaf32196faf9599fa` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:46.921557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] RBAC: Authorizing `identity:get_user(user_id=3e129736df4c49dfaf32196faf9599fa)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:46.924544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:46.928314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 137/263] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/users/3e129736df4c49dfaf32196faf9599fa => generated 283 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:46.938471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:46.951833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 127/264] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 562 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:16:46.957759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:47.156139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4f7b3-8bc8-441d-b173-9f6c0ae94c1d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:47.190587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0eb760-2fa5-44c5-84ab-23d1ed169916 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:47.225402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:47.226389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.226900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.227462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.250197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:47.251077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.251211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.251400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.252539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:47.253989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:47.259514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 128/265] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:47.261886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:47.269241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:47.306596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:47.392682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:47.394193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:47.416295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e586b37a-6181-42d0-8960-01a6ef2ef58a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:47.482617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:47.483190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.483468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.483744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.489984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=3e129736df4c49dfaf32196faf9599fa) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:47.489984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:47.489984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:47.536034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 129/266] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:47 2022] GET /identity/v3/role_assignments?user.id=3e129736df4c49dfaf32196faf9599fa&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:47.575456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 138/267] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 638 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:47.581473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:47.674350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:47.718319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4e79a13-6be2-4993-be36-d89683ceb817 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:47.757078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d171c-8e08-4781-b7da-533a6da67763 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:47.757672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d171c-8e08-4781-b7da-533a6da67763 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.757947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d171c-8e08-4781-b7da-533a6da67763 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.758215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d171c-8e08-4781-b7da-533a6da67763 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.759399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d171c-8e08-4781-b7da-533a6da67763 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:47.760988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d171c-8e08-4781-b7da-533a6da67763 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:47.763225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 130/268] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:47 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:47.772237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:47.871934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:47.922145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2ea0360-d02b-4444-afe7-5afa4690c2ee None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:47.928113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e334b4-9ee2-4003-b235-65c895c15965 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.928670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e334b4-9ee2-4003-b235-65c895c15965 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.929140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0e334b4-9ee2-4003-b235-65c895c15965 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.932796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 131/269] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:47 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:47.938012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eaa891-53f6-4cf6-a6d6-2f1ae9728aca None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.938535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eaa891-53f6-4cf6-a6d6-2f1ae9728aca None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.939036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37eaa891-53f6-4cf6-a6d6-2f1ae9728aca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.970643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-37eaa891-53f6-4cf6-a6d6-2f1ae9728aca None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:47.972195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-37eaa891-53f6-4cf6-a6d6-2f1ae9728aca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:47.974972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:47.976302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:47.976302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:47.976302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:47.976932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] RBAC: Adding query filter params (role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=3e129736df4c49dfaf32196faf9599fa) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:47.977239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:47.978673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:47.985931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 139/270] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:47 2022] GET /identity/v3/role_assignments?user.id=3e129736df4c49dfaf32196faf9599fa&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 608 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:48.055908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 132/271] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:47 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:48.367725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:48.439831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d7773d-899e-419b-be93-7f2873d4fd5e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.440133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d7773d-899e-419b-be93-7f2873d4fd5e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.440394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d7773d-899e-419b-be93-7f2873d4fd5e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.441887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 133/272] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:48.448320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd61b8f-21b9-4547-b748-f9115b8a8bd0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.448641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd61b8f-21b9-4547-b748-f9115b8a8bd0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.448990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd61b8f-21b9-4547-b748-f9115b8a8bd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.480705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2cd61b8f-21b9-4547-b748-f9115b8a8bd0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:48.480705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2cd61b8f-21b9-4547-b748-f9115b8a8bd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:48.497556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b64e4-3aa0-4640-8cb5-754cfc3b5f9f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:48.535138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:48.546227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:48.546903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.547190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.547450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.562337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:48.565918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:48.567499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:48.568373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:48.569155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 140/273] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:48 2022] GET /identity/v3/roles/member => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:48.576437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:48.582809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 134/274] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:48.687892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602efe2-24e8-4635-96c8-f5b9cfa35e95 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:48.723232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97c4fe-2dc0-4d9c-87a7-ed3e4e065017 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.723515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:48.723921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.724196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.724362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97c4fe-2dc0-4d9c-87a7-ed3e4e065017 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.724487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.724657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f97c4fe-2dc0-4d9c-87a7-ed3e4e065017 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.725108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:48.725408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:48.726212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 135/275] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:48.726807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:48.732480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.732862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.733135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.736324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 141/276] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:48 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:48.743945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:48.756456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:48.757627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:48.801276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:48.831775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 136/277] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:48.860434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66bc33bc-dc6c-4979-a6fe-46f265aa1c6f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:48.897230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:48.897860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:48.898142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:48.898417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] PATH_INFO: `/v3/users/717bb30ed8184833a7b967ee157d4f59` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:48.910175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:48.915121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] RBAC: Authorizing `identity:get_user(user_id=717bb30ed8184833a7b967ee157d4f59)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:48.916638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:48.919347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 142/278] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:48 2022] GET /identity/v3/users/717bb30ed8184833a7b967ee157d4f59 => generated 293 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:48.924957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.019103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a94301c5-dca6-43cb-98e9-0392b7360f52 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.036526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-121b36b1-940e-49b9-84fd-499c982bee7b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.048656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:49.049788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:49.061742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.062376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.062823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.063407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.064400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:49.065913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:49.071106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 137/279] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:48 2022] GET /identity/v3/domains/default => generated 199 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:49.073746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.080111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.080963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.081438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.081886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.082810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, user.id=717bb30ed8184833a7b967ee157d4f59, scope.system=all) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:49.083300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:49.084917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:49.091605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 143/280] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Sep 27 16:16:48 2022] GET /identity/v3/role_assignments?user.id=717bb30ed8184833a7b967ee157d4f59&scope.system=all&role.id=1448af55b1904e1199e3954829e9815a => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:49.105824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.192168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4ac816-fda1-49e7-aa58-908bdc24cf88 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.227675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02eb5b10-8650-4bf1-bab5-23c13d3a69fb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.230677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f241561-a48d-4e84-aede-41ef6b83fb7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.231321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f241561-a48d-4e84-aede-41ef6b83fb7b None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.231600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f241561-a48d-4e84-aede-41ef6b83fb7b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.231868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f241561-a48d-4e84-aede-41ef6b83fb7b None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.232527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f241561-a48d-4e84-aede-41ef6b83fb7b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:49.234002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f241561-a48d-4e84-aede-41ef6b83fb7b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:49.266546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0585f5e2-2806-46f9-a530-935f5133ed6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.267210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585f5e2-2806-46f9-a530-935f5133ed6e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.267481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585f5e2-2806-46f9-a530-935f5133ed6e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.267747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0585f5e2-2806-46f9-a530-935f5133ed6e None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.279744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0585f5e2-2806-46f9-a530-935f5133ed6e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-0585f5e2-2806-46f9-a530-935f5133ed6e None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:49.284724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:49.285635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 144/281] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:49 2022] GET /identity/v3/roles/reader => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:49.290215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e31867-5ede-46dd-891c-74f9ad80cc95 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.291196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e31867-5ede-46dd-891c-74f9ad80cc95 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.291196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53e31867-5ede-46dd-891c-74f9ad80cc95 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.291196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 145/282] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:49.294645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e31867-5ede-46dd-891c-74f9ad80cc95 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.331493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:49.335593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 138/283] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:16:49 2022] POST /identity/v3/users => generated 293 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:16:49.345248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.345733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.346331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.372661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:49.374658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:49.422812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53e31867-5ede-46dd-891c-74f9ad80cc95 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.442608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:49.465234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.466186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.466458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.466895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.467738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] RBAC: Adding query filter params (name=reader) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:49.468086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:49.469629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:49.478887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 139/284] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 143 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:49.487044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 146/285] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:49 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:49.487044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.599692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:49.603683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1e9e7-d290-4385-931d-9407652b426e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.642208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.642971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.643249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.643517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] PATH_INFO: `/v3/users/56012eb4ea294d299043849dc08ad010` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.644642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] RBAC: Authorizing `identity:get_user(user_id=56012eb4ea294d299043849dc08ad010)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:49.646069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:49.647969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 147/286] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:49 2022] GET /identity/v3/users/56012eb4ea294d299043849dc08ad010 => generated 297 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:49.653957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.763063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbac7519-da48-4723-ac6f-701b0ed9fe41 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.789424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.806459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:49.807920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:49.808395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:49.808674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:49.809833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:49.811442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:49.818120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 140/287] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:49 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:49.819502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:49.887962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:49.966902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-514e5e78-cab6-450e-8b72-0e632800818b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:49.989431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7186985-c313-4e57-9da1-4bd0659f54ed None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:50.032930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:50.033563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:50.033840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:50.034115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:50.047571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:50.048235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:50.048627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:50.048916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/56012eb4ea294d299043849dc08ad010/roles/29d6af9cebaf4223864fe1061ee89e17` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:50.051008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] RBAC: Authorizing `identity:create_grant(project_id=9c31ce5c991046b0a793e0e05d08cf32, user_id=56012eb4ea294d299043849dc08ad010, role_id=29d6af9cebaf4223864fe1061ee89e17)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:50.052506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:50.058293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:50.064020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:50.070398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:50.071222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 148/288] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 276 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:50.074232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:50.109386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:50.143133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 141/289] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:16:49 2022] PUT /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/users/56012eb4ea294d299043849dc08ad010/roles/29d6af9cebaf4223864fe1061ee89e17 => generated 0 bytes in 324 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:16:50.189754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-135088b5-2977-4381-ab0e-5e585dc7e46b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:50.233443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:50.234076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:50.234356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:50.234626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:50.235411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:50.235716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:50.237163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:50.250793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 149/290] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:50.257096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:50.390968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b201e5-3275-42c8-a803-bf4a6fb35edf None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:50.402703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:50.438836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:50.439485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:50.439759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:50.440030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] PATH_INFO: `/v3/users/25063445b44f41adac6f0ddb96dc9f57` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:50.441166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] RBAC: Authorizing `identity:get_user(user_id=25063445b44f41adac6f0ddb96dc9f57)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:50.442609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:50.444559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 142/291] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:50 2022] GET /identity/v3/users/25063445b44f41adac6f0ddb96dc9f57 => generated 297 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:50.450514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:50.599205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:50.618693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb92f425-f017-456c-aa38-834e6d02bb2b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:50.669098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:50.669626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:50.669902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:50.670170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:50.763324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:50.764861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:50.882939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:50.899645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 150/292] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:50.907877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:51.039743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7676d4d-5b27-4a98-ac73-a73331049afe None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:51.082828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d7cdd91-38dd-4e15-9959-9c961864a647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:51.083740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7cdd91-38dd-4e15-9959-9c961864a647 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:51.084167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7cdd91-38dd-4e15-9959-9c961864a647 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:51.084587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d7cdd91-38dd-4e15-9959-9c961864a647 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:51.085872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7cdd91-38dd-4e15-9959-9c961864a647 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:51.087486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d7cdd91-38dd-4e15-9959-9c961864a647 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:51.089429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 143/293] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:50 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:51.095427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:51.199025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40941c5d-9677-4441-957f-358240fb7fa8 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:51.234600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:51.235492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:51.235920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:51.236337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:51.237200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=25063445b44f41adac6f0ddb96dc9f57) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:51.237654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:51.239263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6e6a8d-788a-40b9-ad42-5ef9d5ac534e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:51.246034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 151/294] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:51 2022] GET /identity/v3/role_assignments?user.id=25063445b44f41adac6f0ddb96dc9f57&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=1448af55b1904e1199e3954829e9815a => generated 608 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:51.469961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766be563-963f-4032-bbcf-53c097e8385c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:51.470476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766be563-963f-4032-bbcf-53c097e8385c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:51.471001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766be563-963f-4032-bbcf-53c097e8385c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:51.472569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 144/295] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:51.479249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:51.479754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:51.480045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:51.502797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:51.506327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:51.554026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:51.580277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 152/296] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:51.880436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766be563-963f-4032-bbcf-53c097e8385c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:51.927267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:51.987982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766be563-963f-4032-bbcf-53c097e8385c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.026161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12885151-3a01-448c-8dde-8ea7b5ed0e5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.026952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12885151-3a01-448c-8dde-8ea7b5ed0e5a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.027230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12885151-3a01-448c-8dde-8ea7b5ed0e5a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.027495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12885151-3a01-448c-8dde-8ea7b5ed0e5a None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.040052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12885151-3a01-448c-8dde-8ea7b5ed0e5a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-12885151-3a01-448c-8dde-8ea7b5ed0e5a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:52.046113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:52.047095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:52.048740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 145/297] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:51 2022] GET /identity/v3/roles/member => generated 84 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:52.053313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:52.141932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d71e33-0ab3-437d-8139-125ed8938387 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.142117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d71e33-0ab3-437d-8139-125ed8938387 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.142380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95d71e33-0ab3-437d-8139-125ed8938387 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.144080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 146/298] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:52.149872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b61ae7-534f-4f4e-9bd7-ca1215e25a73 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.150167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b61ae7-534f-4f4e-9bd7-ca1215e25a73 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.150432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b61ae7-534f-4f4e-9bd7-ca1215e25a73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.166043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b995f425-6c0b-4737-ae21-d470495d32d9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.171407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-84b61ae7-534f-4f4e-9bd7-ca1215e25a73 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:52.172648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-84b61ae7-534f-4f4e-9bd7-ca1215e25a73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:52.210097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.211258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.211681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.212094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.212902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:52.214312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:52.216120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:52.231016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 153/299] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:52 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:52.232412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:52.258251 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 147/300] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:52.280447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:52.286672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6858cd-a2f6-480f-8f00-e0d33b692126 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.287159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6858cd-a2f6-480f-8f00-e0d33b692126 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.287644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6858cd-a2f6-480f-8f00-e0d33b692126 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.289411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 148/301] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:52 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:16:52.295885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.296351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.306857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.316439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:52.317892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:52.354642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e3c14fb-1253-4057-b843-0758b97cb589 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.363129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:52.398640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.399244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.399522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.399793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] PATH_INFO: `/v3/users/717bb30ed8184833a7b967ee157d4f59` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.401524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] RBAC: Authorizing `identity:get_user(user_id=717bb30ed8184833a7b967ee157d4f59)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:52.403602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:52.405921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 154/302] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:52 2022] GET /identity/v3/users/717bb30ed8184833a7b967ee157d4f59 => generated 293 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:52.411145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:52.428574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 149/303] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:52.521154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2f114a3-7e2b-4bd6-a038-7d5e4bb062b7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.532369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:52.551930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:52.564510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.565566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.565846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.566394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] PATH_INFO: `/v3/system/users/717bb30ed8184833a7b967ee157d4f59/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.567977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=717bb30ed8184833a7b967ee157d4f59, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:52.569450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:52.658532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff294655-c465-45dc-b74d-ff6fa39640f6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.675017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 155/304] 10.222.0.64 () {58 vars in 1334 bytes} [Tue Sep 27 16:16:52 2022] PUT /identity/v3/system/users/717bb30ed8184833a7b967ee157d4f59/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:16:52.703942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.704911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.705189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.705500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.707045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:52.708324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:52.710361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 150/305] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:52 2022] GET /identity/v3/domains/default => generated 199 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:52.715927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:52.737793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:52.783439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:52.860040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40c152e-d922-4200-aa08-82f1b2bffe83 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.871741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88c62d92-985c-427d-8de9-a14d4f6984ea None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:52.908265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8db0016e-9081-48dd-8af0-c4b934eade3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.908975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db0016e-9081-48dd-8af0-c4b934eade3d None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.909197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db0016e-9081-48dd-8af0-c4b934eade3d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.909470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db0016e-9081-48dd-8af0-c4b934eade3d None admin] PATH_INFO: `/v3/groups` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.910214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db0016e-9081-48dd-8af0-c4b934eade3d None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:52.911745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8db0016e-9081-48dd-8af0-c4b934eade3d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:52.915145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:52.916035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:52.916678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:52.917265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:52.928351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:52.933576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:52.934467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:52.935321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 151/306] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:52 2022] GET /identity/v3/roles/reader => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:52.940410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:53.022924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 156/307] 10.222.0.64 () {60 vars in 1134 bytes} [Tue Sep 27 16:16:52 2022] POST /identity/v3/groups => generated 229 bytes in 308 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:16:53.024362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c5c61-1bd4-4428-9705-3f454a9c07e5 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.024651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c5c61-1bd4-4428-9705-3f454a9c07e5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.024954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1c5c61-1bd4-4428-9705-3f454a9c07e5 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.027093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 157/308] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:53.032835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.033166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.033742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.054704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:53.059028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:53.060320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:53.069686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6ac48e0-9479-46e4-9d4e-c2ad15888768 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:53.098030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:53.115003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:53.115682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.116020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.116301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.116980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:53.117289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:53.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:53.135297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 152/309] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:52 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:53.138949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:53.154186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 158/310] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:53.258534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e25aa7-0666-4237-b170-db9709737af5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:53.296988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:53.297627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.297905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.298180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] PATH_INFO: `/v3/users/3754487b16154a189704d7d31da6f883` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.313495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] RBAC: Authorizing `identity:get_user(user_id=3754487b16154a189704d7d31da6f883)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:53.314997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:53.316662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 153/311] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:53 2022] GET /identity/v3/users/3754487b16154a189704d7d31da6f883 => generated 293 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:53.322503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:53.436136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a793db61-3285-4bfe-b5b6-b33d4e978dfc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:53.471268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:53.483654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:53.484089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:53.485269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.485551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.485828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.486628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] RBAC: Adding query filter params (role.id=29d6af9cebaf4223864fe1061ee89e17, user.id=3754487b16154a189704d7d31da6f883, scope.system=all) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:53.486967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:53.488478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:53.494165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 159/312] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Sep 27 16:16:53 2022] GET /identity/v3/role_assignments?user.id=3754487b16154a189704d7d31da6f883&scope.system=all&role.id=29d6af9cebaf4223864fe1061ee89e17 => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:53.589681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02efacf4-51f9-4315-9f31-8323940283ca None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:53.627038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:53.627694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.627973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.628241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.641764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:53.647032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:53.648871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:53.650554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 154/313] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:53 2022] GET /identity/v3/roles/reader => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:16:53.656198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:53.690969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:53.766311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4395358c-e880-4d66-a322-172b92f41739 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:53.802715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:53.803686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.804170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.804613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.805561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:53.806037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:53.807725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:53.816692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 160/314] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:53 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:53.822734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:53.899421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:53.929923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9db30b48-5871-4e19-8b7e-6b8ad047a7a4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:53.968027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:53.969022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:53.969477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:53.969921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] PATH_INFO: `/v3/users/56012eb4ea294d299043849dc08ad010` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:53.971266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] RBAC: Authorizing `identity:get_user(user_id=56012eb4ea294d299043849dc08ad010)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:53.972986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:53.975129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 155/315] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:53 2022] GET /identity/v3/users/56012eb4ea294d299043849dc08ad010 => generated 297 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:53.981903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:54.081517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:54.091315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e16f7c78-ca1f-46b3-8efe-b45a7b6a85b9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:54.126305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:54.126944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:54.127227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:54.127499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:54.199153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:54.200933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:54.292125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:54.302592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 161/316] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:53 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:16:54.310124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:54.431205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a92d035-7891-403a-9206-ae9d03c47bd3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:54.467466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-185afd12-f251-4780-805b-e772d50b7118 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:54.468505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185afd12-f251-4780-805b-e772d50b7118 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:54.468998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185afd12-f251-4780-805b-e772d50b7118 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:54.469464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-185afd12-f251-4780-805b-e772d50b7118 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:54.470863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185afd12-f251-4780-805b-e772d50b7118 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:54.472512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-185afd12-f251-4780-805b-e772d50b7118 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:54.474719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 156/317] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:16:54 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:54.484115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:54.589752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d474ad7-bbd1-4d7a-bda6-847df0a6c51a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:54.624303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:54.624910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:54.625185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:54.625455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:54.626179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] RBAC: Adding query filter params (role.id=29d6af9cebaf4223864fe1061ee89e17, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32, user.id=56012eb4ea294d299043849dc08ad010) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:54.626482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:54.627943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-652442fd-ce30-4685-a645-3cb38688586d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:54.634081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 162/318] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:16:54 2022] GET /identity/v3/role_assignments?user.id=56012eb4ea294d299043849dc08ad010&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=29d6af9cebaf4223864fe1061ee89e17 => generated 608 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:55.013489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f6ebf-03b0-4ce9-89be-2db3ddc85b09 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.014000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f6ebf-03b0-4ce9-89be-2db3ddc85b09 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.014405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8f6ebf-03b0-4ce9-89be-2db3ddc85b09 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.017038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 157/319] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:55.023361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f2e5b-ee62-45f0-80a7-f7f1899c104a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.023649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f2e5b-ee62-45f0-80a7-f7f1899c104a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.023925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d44f2e5b-ee62-45f0-80a7-f7f1899c104a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.043771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d44f2e5b-ee62-45f0-80a7-f7f1899c104a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:55.044960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d44f2e5b-ee62-45f0-80a7-f7f1899c104a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:55.097266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:55.114817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 163/320] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:55.279679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d1863-5eff-4f0b-bb2d-3b446fc12f76 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.279915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d1863-5eff-4f0b-bb2d-3b446fc12f76 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.280306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c48d1863-5eff-4f0b-bb2d-3b446fc12f76 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.281782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 158/321] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:55.287064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.287348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.287619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.306735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:55.307953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:55.380444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 164/322] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:55.396090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48d1863-5eff-4f0b-bb2d-3b446fc12f76 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:55.406675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:55.500445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c48d1863-5eff-4f0b-bb2d-3b446fc12f76 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:55.534027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:55.534578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.534886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.535155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.546076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:55.551090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:55.551981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:55.552776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:55.554057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 159/323] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:55.557960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:55.649927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:55.658493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43d16eeb-909b-4645-97e2-fbe692ec30b6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:55.673994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:55.694265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:55.694893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.695170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.695437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.696108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:55.696407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:55.698557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:55.712536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 165/324] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:55 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:55.714461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:55.733729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:55.774079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5cf83-21af-42c8-9f91-c65a4a6a7e80 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:55.811124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:55.811946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.812219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.812489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.814027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:55.815728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:55.817470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 160/325] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:16:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:55.822325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a80232d-bd3b-48e3-865d-d5fb0a25d2e0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:55.823152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:55.855384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bf2faf2-d8af-46a2-af36-0b5d919c4d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:55.856004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf2faf2-d8af-46a2-af36-0b5d919c4d08 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.856279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf2faf2-d8af-46a2-af36-0b5d919c4d08 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.856547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bf2faf2-d8af-46a2-af36-0b5d919c4d08 None admin] PATH_INFO: `/v3/users/717bb30ed8184833a7b967ee157d4f59` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.857648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf2faf2-d8af-46a2-af36-0b5d919c4d08 None admin] RBAC: Authorizing `identity:get_user(user_id=717bb30ed8184833a7b967ee157d4f59)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:55.859103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bf2faf2-d8af-46a2-af36-0b5d919c4d08 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:55.860678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 166/326] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:55 2022] GET /identity/v3/users/717bb30ed8184833a7b967ee157d4f59 => generated 293 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:55.866494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacdfbfc-6c76-4a8a-ac40-dd98e70276f0 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.866794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacdfbfc-6c76-4a8a-ac40-dd98e70276f0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.867067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bacdfbfc-6c76-4a8a-ac40-dd98e70276f0 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.868268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 167/327] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:55.870964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacdfbfc-6c76-4a8a-ac40-dd98e70276f0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:55.915005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:55.922864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b3a5409-f393-4002-b288-f2068583fe35 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:55.955786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:55.956391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:55.956669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:55.956939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] PATH_INFO: `/v3/groups` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:55.957564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:55.959012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:55.962910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:55.970015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bacdfbfc-6c76-4a8a-ac40-dd98e70276f0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:56.002032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:56.002611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:56.002914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:56.003182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:56.003889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] RBAC: Adding query filter params (role.id=1448af55b1904e1199e3954829e9815a, user.id=717bb30ed8184833a7b967ee157d4f59, scope.system=all) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:56.004187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:56.005588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eeff2573-ba97-4523-9646-589702daf8bf None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:56.016775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 168/328] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Sep 27 16:16:55 2022] GET /identity/v3/role_assignments?user.id=717bb30ed8184833a7b967ee157d4f59&scope.system=all&role.id=1448af55b1904e1199e3954829e9815a => generated 510 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:56.018031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:56.018317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:56.018586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:56.038239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:56.039492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:56.071692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 161/329] 10.222.0.64 () {60 vars in 1134 bytes} [Tue Sep 27 16:16:55 2022] POST /identity/v3/groups => generated 226 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:16:56.128566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 169/330] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:56.401645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:56.506056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40e83d6-dc52-49eb-a271-b35a0785a4d4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:56.544592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:56.545239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:56.545519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:56.545790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:56.559338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:56.564701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:56.565564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:56.566386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 162/331] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:56 2022] GET /identity/v3/roles/reader => generated 84 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:56.573228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:56.608924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:56.690933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf2dc45-f3a8-462f-9131-0ef5059a8bfb None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:56.730308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:56.730954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:56.731230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:56.731498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:56.732171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] RBAC: Adding query filter params (name=reader) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:56.732475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:56.733873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:56.744330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 170/332] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:56 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:56.749248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:56.809799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:56.858416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-384f874e-bbf0-4f9d-977f-aedeb39a65d2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:56.894021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfc025b-4b3c-4f6f-8e50-ab9d22bce8d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:56.894581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc025b-4b3c-4f6f-8e50-ab9d22bce8d3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:56.894884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc025b-4b3c-4f6f-8e50-ab9d22bce8d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:56.895158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfc025b-4b3c-4f6f-8e50-ab9d22bce8d3 None admin] PATH_INFO: `/v3/users/3754487b16154a189704d7d31da6f883` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:56.896297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfc025b-4b3c-4f6f-8e50-ab9d22bce8d3 None admin] RBAC: Authorizing `identity:get_user(user_id=3754487b16154a189704d7d31da6f883)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:56.897749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfc025b-4b3c-4f6f-8e50-ab9d22bce8d3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:56.904448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 163/333] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:16:56 2022] GET /identity/v3/users/3754487b16154a189704d7d31da6f883 => generated 293 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:16:56.906381 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:57.022335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:57.032047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c33434-c975-469e-b97d-212ba71ed8e6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:57.073733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995a3d25-002d-4137-a4e0-b433ea004f97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:57.074405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a3d25-002d-4137-a4e0-b433ea004f97 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:57.074685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a3d25-002d-4137-a4e0-b433ea004f97 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:57.074999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995a3d25-002d-4137-a4e0-b433ea004f97 None admin] PATH_INFO: `/v3/system/users/3754487b16154a189704d7d31da6f883/roles/29d6af9cebaf4223864fe1061ee89e17` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:57.078817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995a3d25-002d-4137-a4e0-b433ea004f97 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=3754487b16154a189704d7d31da6f883, role_id=29d6af9cebaf4223864fe1061ee89e17)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:57.081150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-995a3d25-002d-4137-a4e0-b433ea004f97 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:57.153479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 171/334] 10.222.0.64 () {58 vars in 1334 bytes} [Tue Sep 27 16:16:56 2022] PUT /identity/v3/system/users/3754487b16154a189704d7d31da6f883/roles/29d6af9cebaf4223864fe1061ee89e17 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:16:58.733147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58624c45-8015-466a-b9da-317d309c417a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:58.733693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58624c45-8015-466a-b9da-317d309c417a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:58.733693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58624c45-8015-466a-b9da-317d309c417a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:58.735343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 164/335] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:58.740315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:58.740718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:58.740919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:58.772213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:58.774299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:58.821059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6853-08a2-43c9-962c-5b8582c8ad31 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:58.821360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6853-08a2-43c9-962c-5b8582c8ad31 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:58.821612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0d6853-08a2-43c9-962c-5b8582c8ad31 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:58.823270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 165/336] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:58.828330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:58.828610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:58.828883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:58.860147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 172/337] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:16:58.864478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:16:58.865671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:16:58.941269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 166/338] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:16:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:16:59.231653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:59.337154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:59.355251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d72097-377c-40ed-956c-0ef739fa5f87 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:59.361913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:59.420669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:59.421311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:59.421591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:59.421860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:59.447747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:59.453029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:59.455771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:16:59.456572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 173/339] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:16:59 2022] GET /identity/v3/roles/member => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:59.465645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:59.470446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:59.549079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50a825b-186d-47a1-98c8-1f48f4965474 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:59.599674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:59.600588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:59.601010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:59.601420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:59.619887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:16:59.630811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:16:59.631727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:16:59.632518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:16:59.635309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 167/340] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:16:59 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 274 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:16:59.641249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:59.657114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c106fe82-83e7-413c-9e5a-1f7f0c667d66 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:59.737223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:59.738107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:59.738520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:59.738954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:59.739769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:59.740252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:59.741830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:59.763169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 174/341] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:16:59 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 301 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:59.764389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:59.775962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e2689ba-3b6f-43a1-b5c8-3b7974996911 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:16:59.794811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:59.818143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:16:59.857297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:16:59.857915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:59.858196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:59.858467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:59.859165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:16:59.859473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:16:59.861228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab51beb-19c2-4d2f-97b3-23b7e14708ba None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:16:59.886733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 168/342] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:16:59 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:16:59.889668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f959b23-b034-46a7-9da5-8ce4e8df2ed2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:16:59.890946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f959b23-b034-46a7-9da5-8ce4e8df2ed2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:16:59.891369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f959b23-b034-46a7-9da5-8ce4e8df2ed2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:16:59.896616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 169/343] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:16:59 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:16:59.897580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f959b23-b034-46a7-9da5-8ce4e8df2ed2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:16:59.969834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3507b7ff-fa47-4b1d-bd1c-e235a0f98cc4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:00.038883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5119db-366d-4f79-9256-b1188e3d2d98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:00.039505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5119db-366d-4f79-9256-b1188e3d2d98 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.039782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5119db-366d-4f79-9256-b1188e3d2d98 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.040055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5119db-366d-4f79-9256-b1188e3d2d98 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.126626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f959b23-b034-46a7-9da5-8ce4e8df2ed2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:00.180720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5119db-366d-4f79-9256-b1188e3d2d98 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:00.183002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5119db-366d-4f79-9256-b1188e3d2d98 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:00.198727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:00.219543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4798f33-936c-4641-85ca-5b6398f70d2e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:00.220182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4798f33-936c-4641-85ca-5b6398f70d2e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.220468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4798f33-936c-4641-85ca-5b6398f70d2e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.220746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4798f33-936c-4641-85ca-5b6398f70d2e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.315638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 175/344] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 551 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:00.317134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.317627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.318095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.322741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:00.345906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:00.346737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:00.399588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4798f33-936c-4641-85ca-5b6398f70d2e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:00.402381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4798f33-936c-4641-85ca-5b6398f70d2e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:00.431669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 176/345] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:00.433174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:00.484745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:00.578707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 170/346] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:16:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 681 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:00.580235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ff9b16-3eae-42b8-bdda-cbe204a5ca4a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.580517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ff9b16-3eae-42b8-bdda-cbe204a5ca4a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.580787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99ff9b16-3eae-42b8-bdda-cbe204a5ca4a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.582572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 171/347] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:00.585527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7805b249-a241-411d-89d6-ee20480cecd3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:00.587681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ff9b16-3eae-42b8-bdda-cbe204a5ca4a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:00.631694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e340cc3-7268-462d-9231-49e0c228639b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:00.633274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e340cc3-7268-462d-9231-49e0c228639b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.634021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e340cc3-7268-462d-9231-49e0c228639b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.634544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e340cc3-7268-462d-9231-49e0c228639b None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.637243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e340cc3-7268-462d-9231-49e0c228639b None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:00.642787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e340cc3-7268-462d-9231-49e0c228639b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:00.645046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 177/348] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:00 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:00.646637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.647085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.647678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.675285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:00.676849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:00.738229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:00.771992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 178/349] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:00.773424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:00.830142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:00.843470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99ff9b16-3eae-42b8-bdda-cbe204a5ca4a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:00.918193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:00.919512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:00.919937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:00.920351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:00.921684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:00.923487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:00.925940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 172/350] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:00 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 338 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:00.927316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:00.981893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cbea0b-53c3-4194-b49b-e05c269b35bf None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.051591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-649647d9-e237-4b1f-9994-48460c6c0320 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.072118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.073296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.073982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.075393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.090458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:01.095546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:01.099611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 179/351] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:00 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 325 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:01.100752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.132677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.135713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.138069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.138552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.160977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:01.169596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:01.170715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:01.171719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:01.174513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 173/352] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 248 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:01.175536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.179052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:01.277521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d715482-c49c-40ba-b37e-d4fa39653f2a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.279662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:01.300573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54207bdb-84ff-4ab1-adc2-7d012df84e55 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.317869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.318505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.318798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.319072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.320166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:01.321615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:01.328769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 180/353] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:01.329929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.341803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.341803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.341803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.341803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.342226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:01.342226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:01.343268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:01.349891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 174/354] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=1448af55b1904e1199e3954829e9815a => generated 260 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:01.351026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.449471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:01.482763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85314ddc-6456-436b-b9c4-dc92fff0dd29 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.503169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67686ae-2071-4406-b900-01614c5c8241 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.537452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.538559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.539093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.539619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.540518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:01.541650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:01.544107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2998f29-567e-42c3-93bc-1ca3bbd675e2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:01.545114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.545759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.546039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.546311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.555573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 181/355] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:01.558154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992530c6-19f1-44b3-9f8c-344be1debffb None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.558436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992530c6-19f1-44b3-9f8c-344be1debffb None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.558710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992530c6-19f1-44b3-9f8c-344be1debffb None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.559666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:01.561574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 182/356] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:01 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:01.562666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060595eb-a220-417d-82d6-71a23694ca0a None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.562974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060595eb-a220-417d-82d6-71a23694ca0a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.563275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-060595eb-a220-417d-82d6-71a23694ca0a None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.566481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 183/357] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:01 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:01.566481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060595eb-a220-417d-82d6-71a23694ca0a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:01.566677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:01.567587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 175/358] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 217 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:01.568384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.613333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:01.691429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d78da412-a4c0-4730-a403-5f5d958de814 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.692522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-060595eb-a220-417d-82d6-71a23694ca0a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:01.731823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:01.736247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.736848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.737124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.746305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.746305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:01.746305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:01.746305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29620a68-0b7b-493b-8566-7f63a2946b60 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:01.746591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612b6dd5-fe90-470e-b0d0-961b15c98117 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:01.746591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b6dd5-fe90-470e-b0d0-961b15c98117 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.746591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b6dd5-fe90-470e-b0d0-961b15c98117 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.746591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612b6dd5-fe90-470e-b0d0-961b15c98117 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.747832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 184/359] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 260 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:01.749296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.749629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.749901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:01.778535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:01.779854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:01.852987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b6dd5-fe90-470e-b0d0-961b15c98117 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:01.855851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612b6dd5-fe90-470e-b0d0-961b15c98117 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:01.882439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 185/360] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:01.883507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:01.918965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:01.978266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 176/361] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 410 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:01.980010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:01.980431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:01.980837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.002376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:02.003867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:02.004514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defafbf1-9184-4839-96d8-1b13c8cfd3f9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.027439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.046010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.046759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.047136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.047483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.048227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:02.048733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:02.050360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:02.061774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 186/362] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:02.062788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.084864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 177/363] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:02.085964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.175944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.190151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b41fa8b-a915-48a8-9bcd-7cad414bc9d5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.207540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8726fffe-b7d9-419d-bf05-cfc2ce9cb62d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.227856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.229152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.229615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.230514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.231356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:02.232794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:02.234856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 187/364] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:02.240324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.257128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.258106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.258592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.259202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.287678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.335700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:02.337508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:02.449457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5c36a69-493c-457e-880c-a161fc6c7a86 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.450126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 178/365] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:02.456935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.491472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.492109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.492389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.492660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.497905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:02.499437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:02.501422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 188/366] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:02.502277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.532382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.547556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.597040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9eecc07-3529-4b27-99c2-b1e08e3cc94d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.620938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e4d9515-99f4-48f0-b273-0d2e44a6cebc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.641158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.641904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.642187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.642457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.643628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:02.645326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:02.650983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 179/367] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:02.651937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.656551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.657187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.657466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.657735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] PATH_INFO: `/v3/roles/reader` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.670499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:02.676907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:02.677815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:02.678623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 189/368] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/roles/reader => generated 84 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:02.679346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.770106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a378c5-c01e-499a-85e9-0ff7e5442219 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.773527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.792655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614b8747-57b4-4d6f-b5e1-3b2e89442c1f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.796711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:02.814078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.814710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.815019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.815292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.816125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:02.816442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:02.818006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:02.825969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 180/369] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=1448af55b1904e1199e3954829e9815a => generated 260 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:02.826703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.828882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:02.829853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:02.830128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:02.830395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:02.842718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:02.849351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:02.851472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 190/370] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:02.852975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:02.955151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d47e9d74-2891-4b73-92c1-783580431dae None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:02.974495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e10f62-68b8-4328-8d18-5ada32b7011d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.012624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.013479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.013696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.014099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.014279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.014379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.014634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.014926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.015342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] RBAC: Adding query filter params (name=reader) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:03.015648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.017046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.017821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.019689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.021565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 181/371] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:03.022605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:03.028790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 191/372] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:17:02 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:03.029812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:03.039849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:03.117201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:03.143816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120b808-a92e-4e47-91cb-96f3562e3b8f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.180453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-427df8d0-0a54-453e-a764-b9a3e2807604 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.189980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.190601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.190931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.191217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.192043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:03.192378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.193816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.206860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 192/373] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:03 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:03.207703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:03.247116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.247753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.248031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.248309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.249939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:03.250250 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.251697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.265521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 182/374] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:03.275658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:03.332129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:03.395734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8dfbb3-7905-4ca8-8d52-d58f083ef306 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.461518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.462165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.462448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.463367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] PATH_INFO: `/v3/users/3754487b16154a189704d7d31da6f883` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.465850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] RBAC: Authorizing `identity:get_user(user_id=3754487b16154a189704d7d31da6f883)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.467336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.470063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 193/375] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:17:03 2022] GET /identity/v3/users/3754487b16154a189704d7d31da6f883 => generated 293 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:03.474960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:03.477729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b7c273-07c7-4dc2-a436-d2c26c2854d7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.504393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:03.550705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.552031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.552546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.552820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.667331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.672244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.676733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bf72d2d-6a68-4fd3-9e40-b5cda2cd0fba None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.776376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.777012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.777295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.777575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:03.778317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] RBAC: Adding query filter params (role.id=29d6af9cebaf4223864fe1061ee89e17, user.id=3754487b16154a189704d7d31da6f883, scope.system=all) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:03.778645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:03.780109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:03.792512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:03.798344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 194/376] 10.222.0.64 () {56 vars in 1308 bytes} [Tue Sep 27 16:17:03 2022] GET /identity/v3/role_assignments?user.id=3754487b16154a189704d7d31da6f883&scope.system=all&role.id=29d6af9cebaf4223864fe1061ee89e17 => generated 510 bytes in 324 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:03.824629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 183/377] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 549 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:03.830199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:03.956735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6214f31-4f49-4107-bbde-98a8cb387ad4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:03.997088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:03.997732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:03.998012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:03.998285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] PATH_INFO: `/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.003681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] RBAC: Authorizing `identity:get_project(project_id=9ac75ab2f7e84b088f06cdcdbcfae272)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:04.005168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:04.007294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 195/378] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:03 2022] GET /identity/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272 => generated 344 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:04.012929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:04.047705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:04.161390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29c54f7d-cb12-48da-8369-71744b32138f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:04.199802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f22aa12-875c-494e-805f-39e4d9364fd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:04.200498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f22aa12-875c-494e-805f-39e4d9364fd9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:04.200782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f22aa12-875c-494e-805f-39e4d9364fd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:04.201055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f22aa12-875c-494e-805f-39e4d9364fd9 None admin] PATH_INFO: `/v3/groups/c03b057d0fae443d813860edace6d994` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.209879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f22aa12-875c-494e-805f-39e4d9364fd9 None admin] RBAC: Authorizing `identity:get_group(group_id=c03b057d0fae443d813860edace6d994)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:04.211367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f22aa12-875c-494e-805f-39e4d9364fd9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:04.214064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 184/379] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:04 2022] GET /identity/v3/groups/c03b057d0fae443d813860edace6d994 => generated 229 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:04.219817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:04.305440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:04.348795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91c29305-48bf-49ea-8f3f-1363a018ec0c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:04.394433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:04.395077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:04.395361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:04.395639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.396368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] RBAC: Adding query filter params (group.id=c03b057d0fae443d813860edace6d994, role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=9ac75ab2f7e84b088f06cdcdbcfae272) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:04.396679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:04.398115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ca3e08a-5e43-4c11-a4b3-931919e76872 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:04.404909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 196/380] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:04 2022] GET /identity/v3/role_assignments?group.id=c03b057d0fae443d813860edace6d994&scope.project.id=9ac75ab2f7e84b088f06cdcdbcfae272&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 260 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:04.759269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb274b7-15cb-409a-b5f4-55966ca08d02 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:04.760076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb274b7-15cb-409a-b5f4-55966ca08d02 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:04.760362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fb274b7-15cb-409a-b5f4-55966ca08d02 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.761977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 185/381] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:04.767230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:04.767823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:04.768311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.788834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:04.790405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:04.852304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d25b5e-fa9a-4378-b68e-c8cc1e27fcb7 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:04.852946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d25b5e-fa9a-4378-b68e-c8cc1e27fcb7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:04.853981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3d25b5e-fa9a-4378-b68e-c8cc1e27fcb7 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.855657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 186/382] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:04.881722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 197/383] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:04.883883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:04.884426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:04.885473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:04.930815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:04.932538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:05.036471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 187/384] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:05.190088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:05.270064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:05.336364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3aee55f-a1f9-480b-96ac-59af5d4398b4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:05.386614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:05.387322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:05.387603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:05.387872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:05.398655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:05.408492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:05.409630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:05.410620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 198/385] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:05 2022] GET /identity/v3/roles/member => generated 84 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:05.416413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:05.432297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:05.476120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:05.576415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5633610a-ae75-450d-9380-e6c57ab27622 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:05.657907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:05.658675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:05.659091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:05.659413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:05.660080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:05.660383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:05.661949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:05.664829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d92d29d-d540-4de5-a26d-4a6f03792b2c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:05.679867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 188/386] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:17:05 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:05.692188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:05.693491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:05.718038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:05.718674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:05.719017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:05.719286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:05.739225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:05.744716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:05.745572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:05.746359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:05.747360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 199/387] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:17:05 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:05.758196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:05.837449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:05.889058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938a26f9-5741-432b-a824-1a9ed5ecdb09 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:05.895013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6ef99c2-e7a9-460b-80b6-80474035a6e2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:05.937204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:05.946975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:05.946975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:05.946975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:05.946975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:05.946975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:05.946975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:05.959817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 200/388] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:05 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:05.975599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:05.981157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:05.983924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:05.984204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:05.984478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:05.986564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:05.991413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:05.996999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 189/389] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:05 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 304 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:06.002138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:06.091184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:06.201236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f62e861-5451-4266-a3d2-ae16daf80255 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:06.216752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda4a58c-5588-46e9-8460-a842f1100051 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:06.248037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3421578-c516-4622-906d-07dc57e7d1e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:06.248682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3421578-c516-4622-906d-07dc57e7d1e3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.248955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3421578-c516-4622-906d-07dc57e7d1e3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.249887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3421578-c516-4622-906d-07dc57e7d1e3 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.250355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3421578-c516-4622-906d-07dc57e7d1e3 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:06.258800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3421578-c516-4622-906d-07dc57e7d1e3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:06.263189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 201/390] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:05 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 287 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:06.266921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f6624-cac2-4b4b-9ad2-f215581bd06b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.266921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f6624-cac2-4b4b-9ad2-f215581bd06b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.266921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f18f6624-cac2-4b4b-9ad2-f215581bd06b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.270800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 202/391] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:06 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:06.270800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f6624-cac2-4b4b-9ad2-f215581bd06b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:06.291211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89f5c7bf-a493-485d-b31e-031d0e0e90be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:06.291211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5c7bf-a493-485d-b31e-031d0e0e90be None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.291211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5c7bf-a493-485d-b31e-031d0e0e90be None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.291211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89f5c7bf-a493-485d-b31e-031d0e0e90be None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.295123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f5c7bf-a493-485d-b31e-031d0e0e90be None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:06.295123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89f5c7bf-a493-485d-b31e-031d0e0e90be None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:06.296302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 190/392] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:06 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 295 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:06.302028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.302310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.302588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.332103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:06.332103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:06.338204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:06.448935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f18f6624-cac2-4b4b-9ad2-f215581bd06b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:06.466444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:06.496537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 191/393] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:06.497471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:06.527762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:06.528427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.528707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.528982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.532711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:06.534542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:06.536545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 203/394] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:06 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:06.542656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:06.672253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bdf7aba-6b47-4d02-84ca-3ec337255ecb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:06.698813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57521115-0740-4cc6-a2bc-2358d2197cdc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:06.708057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2799c0a2-eb09-48a2-8462-9a19965ff6fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:06.708573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799c0a2-eb09-48a2-8462-9a19965ff6fd None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.708847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799c0a2-eb09-48a2-8462-9a19965ff6fd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.709116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2799c0a2-eb09-48a2-8462-9a19965ff6fd None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.711021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799c0a2-eb09-48a2-8462-9a19965ff6fd None admin] RBAC: Authorizing `identity:create_grant(project_id=1d0285a452bf4bb285cfc1f65cacd3ac, group_id=e8608d7a32114df5aaebe402ba91ecf1, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:06.713031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2799c0a2-eb09-48a2-8462-9a19965ff6fd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:06.737677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:06.738347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.738698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.739079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.741310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] RBAC: Authorizing `identity:create_grant(project_id=1d0285a452bf4bb285cfc1f65cacd3ac, group_id=e8608d7a32114df5aaebe402ba91ecf1, role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:06.742861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:06.781424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:06.786031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 192/395] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Sep 27 16:17:06 2022] PUT /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:06.787646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444b3d7b-28d5-4336-a935-54fb46284163 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.788136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444b3d7b-28d5-4336-a935-54fb46284163 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.788607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-444b3d7b-28d5-4336-a935-54fb46284163 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.790175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 193/396] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:06.797078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.797709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.798197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.818569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:06.820339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:06.824578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 204/397] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Sep 27 16:17:06 2022] PUT /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 0 bytes in 282 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:06.830926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:06.868521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:06.944235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc660a6b-5b43-4e5e-8c69-8fbc0b128254 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:06.959212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 194/398] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:06.984979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887cb29f-916f-4f4a-896d-c2912fb3011c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:06.985769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887cb29f-916f-4f4a-896d-c2912fb3011c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:06.986121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887cb29f-916f-4f4a-896d-c2912fb3011c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:06.986466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887cb29f-916f-4f4a-896d-c2912fb3011c None admin] PATH_INFO: `/v3/roles/member` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:06.999655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887cb29f-916f-4f4a-896d-c2912fb3011c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-887cb29f-916f-4f4a-896d-c2912fb3011c None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:07.005391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:07.006291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:07.007118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 205/399] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:06 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:07.016532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.160677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d113b24d-9b4e-41ef-aac4-f764e683b6e0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.190485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d5c5d-20a7-4326-a437-7e8a10cd395b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.190765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d5c5d-20a7-4326-a437-7e8a10cd395b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.191064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61d5c5d-20a7-4326-a437-7e8a10cd395b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.192652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 206/400] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:07.195046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:07.199487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.199770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.201779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.202027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.202825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.203106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.203383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.204290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] RBAC: Adding query filter params (name=member) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:07.204591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:07.205957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:07.215629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 195/401] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:07.224086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.233303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:07.234525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:07.351297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 207/402] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:07.352237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.358616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76945969-088b-4abb-b604-aa504347f6e4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.382685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:07.404340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.405585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.406391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.406833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.409303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:07.410916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:07.413759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 196/403] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:07.419679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.469531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0be7e0be-0ab2-48b9-82f2-5f76236fb883 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.481802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:07.514680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.515344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.515616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.515885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.525212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b39e089f-beda-41b3-b0b9-4214ccae9664 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.536283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:07.547234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:07.548350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:07.549196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 208/404] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:07.554623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.564162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.564990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.565455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.565866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.567182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:07.568751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:07.570508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 197/405] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:07.576971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.702146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:07.718329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c41df98-090f-491a-903b-e391ae78392e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.727290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f29a7de-6cc8-4af9-b957-3deac48fd029 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.753602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.754206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.754483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.754751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.755435 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:07.755732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:07.757102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:07.763969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:07.770824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 209/406] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:07.770824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.780163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.781124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.781547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.781982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.784991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] RBAC: Authorizing `identity:create_grant(project_id=9c31ce5c991046b0a793e0e05d08cf32, group_id=e8608d7a32114df5aaebe402ba91ecf1, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:07.786969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:07.875518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f105c4a9-99ae-4089-9dde-d52b0389ba1a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:07.888170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 198/407] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Sep 27 16:17:07 2022] PUT /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:07.896529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.938423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:07.939213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:07.939491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:07.939761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:07.958894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:07.966818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:07.967767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:07.968567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 210/408] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/roles/admin => generated 83 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:07.973739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:07.994727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:08.112775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ed4009-0f52-424c-98c2-ff1023765080 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.177750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.178424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.178709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.179157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.180973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.182718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:08.186191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fb92b3-1e5e-47e2-b8b5-44cc95658c70 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.189587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 199/409] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:08.192682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:08.236203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.236770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.237050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.237321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.237992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:08.238293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.239925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:08.254802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 211/410] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:07 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:08.258305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:08.280271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:08.338090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:08.373525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920ecd88-ae1e-4e0b-9a7c-3d10be1dc237 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.392633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556de45-62c5-4203-a4d7-30c292d5f2d2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.445426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.446336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.447600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.449368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] PATH_INFO: `/v3/groups/c03b057d0fae443d813860edace6d994` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.451350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] RBAC: Authorizing `identity:get_group(group_id=c03b057d0fae443d813860edace6d994)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.453502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:08.456914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 212/411] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:08 2022] GET /identity/v3/groups/c03b057d0fae443d813860edace6d994 => generated 229 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:08.464628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:08.466947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.469769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.470210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.470645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.473594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.480464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:08.482662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 200/412] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:08 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:08.493593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:08.660043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcecf46c-ded1-4b77-8dc3-bec58cbe2c25 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.674174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb277140-7cd4-4a93-910c-ba3d52caf15a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.708046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:08.728866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2854fe4f-a7af-4728-be23-2de7e18c51ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.729786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854fe4f-a7af-4728-be23-2de7e18c51ca None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.730073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854fe4f-a7af-4728-be23-2de7e18c51ca None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.730355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2854fe4f-a7af-4728-be23-2de7e18c51ca None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.733898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854fe4f-a7af-4728-be23-2de7e18c51ca None admin] RBAC: Authorizing `identity:create_grant(project_id=9c31ce5c991046b0a793e0e05d08cf32, group_id=e8608d7a32114df5aaebe402ba91ecf1, role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.735867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2854fe4f-a7af-4728-be23-2de7e18c51ca None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:08.742314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.751095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.751095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.751095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] PATH_INFO: `/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.751095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] RBAC: Authorizing `identity:get_project(project_id=9ac75ab2f7e84b088f06cdcdbcfae272)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.751095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:08.751095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 213/413] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:08 2022] GET /identity/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272 => generated 344 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:08.755515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:08.766715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:08.839854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 201/414] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Sep 27 16:17:08 2022] PUT /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32/groups/e8608d7a32114df5aaebe402ba91ecf1/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:17:08.883006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46cec2c4-ed46-4004-879f-06605aef9e18 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:08.928870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da4dd9d-7371-4e61-ac4c-c0025a6e799c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:08.929586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da4dd9d-7371-4e61-ac4c-c0025a6e799c None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:08.929880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da4dd9d-7371-4e61-ac4c-c0025a6e799c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:08.930157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da4dd9d-7371-4e61-ac4c-c0025a6e799c None admin] PATH_INFO: `/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272/groups/c03b057d0fae443d813860edace6d994/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:08.934269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da4dd9d-7371-4e61-ac4c-c0025a6e799c None admin] RBAC: Authorizing `identity:create_grant(project_id=9ac75ab2f7e84b088f06cdcdbcfae272, group_id=c03b057d0fae443d813860edace6d994, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:08.935759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da4dd9d-7371-4e61-ac4c-c0025a6e799c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:09.030078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 214/415] 10.222.0.64 () {58 vars in 1442 bytes} [Tue Sep 27 16:17:08 2022] PUT /identity/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272/groups/c03b057d0fae443d813860edace6d994/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:17:09.276484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d407bfd-099f-40c0-8474-8a73b80414d7 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:09.276869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d407bfd-099f-40c0-8474-8a73b80414d7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:09.277045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d407bfd-099f-40c0-8474-8a73b80414d7 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:09.278662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 202/416] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:09 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:09.284270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7053fe5-595f-4f64-a370-c5a1a4c4bd75 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:09.284504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7053fe5-595f-4f64-a370-c5a1a4c4bd75 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:09.284773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7053fe5-595f-4f64-a370-c5a1a4c4bd75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:09.303408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c7053fe5-595f-4f64-a370-c5a1a4c4bd75 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:09.304613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c7053fe5-595f-4f64-a370-c5a1a4c4bd75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:09.378918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:09.392899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 215/417] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:09.645219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc3307-9e4d-46ce-a6c0-cb9bbd37730b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:09.645730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc3307-9e4d-46ce-a6c0-cb9bbd37730b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:09.645833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07bc3307-9e4d-46ce-a6c0-cb9bbd37730b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:09.647449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 203/418] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:09.653018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:09.653311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:09.653589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:09.679695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:09.680892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:09.767093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 216/419] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:09.876253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc3307-9e4d-46ce-a6c0-cb9bbd37730b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:09.990026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07bc3307-9e4d-46ce-a6c0-cb9bbd37730b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.030519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:10.031204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.031487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.031764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.044187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:10.049352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:10.050201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:10.051026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 204/420] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:09 2022] GET /identity/v3/roles/member => generated 84 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:10.060345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:10.207617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07d0596-e85b-4f97-a3fe-5dda1e79741a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.247160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:10.259004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:10.259367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:10.262688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.262688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.262688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.262688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:10.262688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:10.263205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:10.271933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 217/421] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:17:10 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:10.278544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:10.349577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:10.421243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d35b4ff-78c6-4089-b3da-21523b68666c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.470662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257cf904-f40a-4a45-8022-73e0e6d21ffd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:10.471539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257cf904-f40a-4a45-8022-73e0e6d21ffd None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.472003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257cf904-f40a-4a45-8022-73e0e6d21ffd None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.472416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257cf904-f40a-4a45-8022-73e0e6d21ffd None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.496998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5557ea72-e378-4699-ae81-061973f04df4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.553846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5227e219-4484-403b-9741-1c1869053d0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:10.554822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5227e219-4484-403b-9741-1c1869053d0e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.555273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5227e219-4484-403b-9741-1c1869053d0e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.555717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5227e219-4484-403b-9741-1c1869053d0e None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.572651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:10.573022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5227e219-4484-403b-9741-1c1869053d0e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5227e219-4484-403b-9741-1c1869053d0e None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:10.580577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:10.581466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:10.582279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:10.589118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 205/422] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:17:10 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:10.589118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5098f-f3a7-43b9-916a-2d1bd1bad47e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.589118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5098f-f3a7-43b9-916a-2d1bd1bad47e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.589118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad5098f-f3a7-43b9-916a-2d1bd1bad47e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.589917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 206/423] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:10.590903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257cf904-f40a-4a45-8022-73e0e6d21ffd None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:10.591209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad5098f-f3a7-43b9-916a-2d1bd1bad47e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:10.592620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257cf904-f40a-4a45-8022-73e0e6d21ffd None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:10.711128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad5098f-f3a7-43b9-916a-2d1bd1bad47e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.711859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 218/424] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:17:10.713893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.714343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.714679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.715699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:10.733986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:10.735456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:10.749991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:10.750880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.751271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.751844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:10.752526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:10.752931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:10.754545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:10.764193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 207/425] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:10 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:10.765314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:10.812137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 219/426] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:10.813409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:10.834377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:10.937357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-235f5272-b421-4e81-9d91-d6df94afe54f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.988087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e2e22b-ebbd-438b-ade8-b00d7989e12a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:10.998091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:10.998730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:10.999028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:10.999297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.001511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.003040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.006718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 208/427] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:10 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:11.010443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.053717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.054328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.054657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.054948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.075636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.150674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-698ac212-1c86-4ee6-93c6-6674065e9538 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.178354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.180970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.189913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.190666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.191410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.191410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.192714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.194444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.196542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 209/428] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:11.197516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.204357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.308676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f233039-8406-4f8e-b705-895d05cf7c47 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.310554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.313683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 220/429] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:10 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 500 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:11.314759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.357702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.358467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.358865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.359238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] PATH_INFO: `/v3/roles/member` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.371154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:11.376645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:11.377605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:11.378403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 210/430] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:11.379096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.442259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83dfa6ad-25c6-48d6-942f-7f3aac8ac870 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.471153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.492057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.492957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.493239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.493539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.494304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:11.494852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.497600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.499734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff10d68a-89ef-4c9f-8863-9d8ce1d0624e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.506020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 221/431] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=1448af55b1904e1199e3954829e9815a => generated 611 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:11.506951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.553159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1373eaac-e317-4d17-8b97-97f8cb349004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.554076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1373eaac-e317-4d17-8b97-97f8cb349004 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.554353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1373eaac-e317-4d17-8b97-97f8cb349004 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.554916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1373eaac-e317-4d17-8b97-97f8cb349004 None admin] PATH_INFO: `/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.556834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1373eaac-e317-4d17-8b97-97f8cb349004 None admin] RBAC: Authorizing `identity:get_project(project_id=1d0285a452bf4bb285cfc1f65cacd3ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.558704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1373eaac-e317-4d17-8b97-97f8cb349004 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.561593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 211/432] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/projects/1d0285a452bf4bb285cfc1f65cacd3ac => generated 298 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:11.562977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d7737-7840-465c-b27f-aae566c94285 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.563252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d7737-7840-465c-b27f-aae566c94285 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.563523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1d7737-7840-465c-b27f-aae566c94285 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.565007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 212/433] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:11.566537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1d7737-7840-465c-b27f-aae566c94285 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.612058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.657574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdaacb19-8725-467a-ac0a-b9f708fad45b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.696215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.697382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.697653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.697920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.698590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] RBAC: Adding query filter params (name=member) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:11.698917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.700284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d987dbd7-02a5-4f2c-bb90-b4de0a9b67a5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.702639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1d7737-7840-465c-b27f-aae566c94285 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.715489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 222/434] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:11.718640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.718640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.718640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.736839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:11.738035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:11.742534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9291fe64-3b07-42d8-b528-4aaa05f39400 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.743218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9291fe64-3b07-42d8-b528-4aaa05f39400 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.744019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9291fe64-3b07-42d8-b528-4aaa05f39400 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.745831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9291fe64-3b07-42d8-b528-4aaa05f39400 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.745831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9291fe64-3b07-42d8-b528-4aaa05f39400 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:11.747023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9291fe64-3b07-42d8-b528-4aaa05f39400 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:11.748621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 213/435] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:11.749837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e681460-1192-457e-8b63-462af0aa3e17 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.750124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e681460-1192-457e-8b63-462af0aa3e17 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.750400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e681460-1192-457e-8b63-462af0aa3e17 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.751647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 214/436] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:11 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:11.752347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e681460-1192-457e-8b63-462af0aa3e17 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.788112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.819961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 223/437] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:11.819961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:11.874645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:11.879882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e681460-1192-457e-8b63-462af0aa3e17 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:11.925487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:11.926109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:11.926389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:11.926661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:11.960935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120e24b-8415-4f6f-8a98-89bcd869feb2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.010622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.011266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.011539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.011968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.012694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:12.013497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.014968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43ff02cf-e735-4383-bc79-8174b3fd23a3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.015137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.016565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.024007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 224/438] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=1d0285a452bf4bb285cfc1f65cacd3ac&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 611 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:12.025654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.025933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.026200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.048004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:12.049340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:12.087048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.142894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 225/439] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:12.142894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.166018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.176619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 215/440] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 424 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:12.177936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.245664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c8f115-645d-41bf-9cc2-7cd18c7971a2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.289200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.289840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.290117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.290935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.291037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c120a5aa-1522-4c5d-a85c-c6933ee81244 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.303028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:12.312339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:12.313245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:12.314101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 226/441] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:12.319728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.339369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.340411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.340834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.341304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.342753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.345319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.352063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 216/442] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:12.352576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.356811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.435823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.450190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a2f2ca5-deb7-418c-9f4d-ffa36a0944a8 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.474988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e26be751-f428-4f05-9da3-34b7eccd5fae None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.491141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.491761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.492037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.492644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.492962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:12.493261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.495342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.504408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 227/443] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:12.505263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.528061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.528707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.528991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.529714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.530419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.531938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.533828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 217/444] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:12.536556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.605443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.614608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3b79-6cae-4c5e-aad6-678504144a72 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.638005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e486b07b-274d-4dd3-915e-6d01e771f18a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.649001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.649650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.650051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.650532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.660604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:12.666046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:12.666967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:12.667761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 228/445] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:12.668475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.673576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.674188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.674476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.674750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.677441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.743017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.745430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.768889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74534c77-e58a-4493-a41b-e46e9160622e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.800609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.801201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.801474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.801742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.802456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=1448af55b1904e1199e3954829e9815a, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:12.802765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.804198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.810308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 229/446] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=1448af55b1904e1199e3954829e9815a => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:12.811110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.828267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.872030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 218/447] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 337 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:12.877926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:12.914144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:12.938692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae38ffd-8987-452f-969f-d940570ca79f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.976154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:12.976771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:12.977044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:12.977310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:12.978510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:12.978836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:12.980339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:12.989237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 230/448] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:12.993266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99945360-ff23-4f43-88be-8552bb7607e9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:12.995677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:13.034245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.034911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.035189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.035461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] PATH_INFO: `/v3/projects/9c31ce5c991046b0a793e0e05d08cf32` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.036564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] RBAC: Authorizing `identity:get_project(project_id=9c31ce5c991046b0a793e0e05d08cf32)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.037992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.040941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 219/449] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/projects/9c31ce5c991046b0a793e0e05d08cf32 => generated 302 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:13.045738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:13.097150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:13.125367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2eb3b2-c4e9-463e-b964-bf739cb55681 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:13.161221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54dfc753-0aee-4cce-b7f4-fe23492189f0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:13.162847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:13.169172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.170186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.170347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.170559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.208010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.208763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.209157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.209436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] PATH_INFO: `/v3/groups/e8608d7a32114df5aaebe402ba91ecf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.211540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] RBAC: Authorizing `identity:get_group(group_id=e8608d7a32114df5aaebe402ba91ecf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.212040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.215044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 220/450] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:13 2022] GET /identity/v3/groups/e8608d7a32114df5aaebe402ba91ecf1 => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:13.220570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:13.249178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.250570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.317234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-524d8603-7d1d-4a20-a0f2-7b9972c536c4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:13.325271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:13.350198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 231/451] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:12 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 355 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:17:13.354233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.354958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.355310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.355591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.356305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] RBAC: Adding query filter params (group.id=e8608d7a32114df5aaebe402ba91ecf1, role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=9c31ce5c991046b0a793e0e05d08cf32) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:13.356608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.356715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:13.358025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.363834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 221/452] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:13 2022] GET /identity/v3/role_assignments?group.id=e8608d7a32114df5aaebe402ba91ecf1&scope.project.id=9c31ce5c991046b0a793e0e05d08cf32&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 611 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:13.469048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15131e47-22dd-4c07-9fbc-14b332c25de9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:13.504800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.505358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.505634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.505903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] PATH_INFO: `/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.507135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] RBAC: Authorizing `identity:get_project(project_id=9ac75ab2f7e84b088f06cdcdbcfae272)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.508597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.510453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 232/453] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:17:13 2022] GET /identity/v3/projects/9ac75ab2f7e84b088f06cdcdbcfae272 => generated 344 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:13.516153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:13.561224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:13.630144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e85cd30-67e3-47bc-b6ac-faf45df98e6c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:13.664785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c36aa476-57d8-462f-84cd-79327270650c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.665536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36aa476-57d8-462f-84cd-79327270650c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.665817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36aa476-57d8-462f-84cd-79327270650c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.666091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c36aa476-57d8-462f-84cd-79327270650c None admin] PATH_INFO: `/v3/groups/c03b057d0fae443d813860edace6d994` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.667220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36aa476-57d8-462f-84cd-79327270650c None admin] RBAC: Authorizing `identity:get_group(group_id=c03b057d0fae443d813860edace6d994)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.668666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c36aa476-57d8-462f-84cd-79327270650c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.670226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 222/454] 10.222.0.64 () {56 vars in 1180 bytes} [Tue Sep 27 16:17:13 2022] GET /identity/v3/groups/c03b057d0fae443d813860edace6d994 => generated 229 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:13.676299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:13.724402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:13.778523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d80e2af-bbdd-4aec-b6fe-c6abe39feee8 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:13.810057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:13.810653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:13.810961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:13.811232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:13.811944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] RBAC: Adding query filter params (group.id=c03b057d0fae443d813860edace6d994, role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=9ac75ab2f7e84b088f06cdcdbcfae272) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:13.812246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:13.813669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ced9bd92-2331-4425-b1dc-3f81ac90d764 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:13.819160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 233/455] 10.222.0.64 () {56 vars in 1376 bytes} [Tue Sep 27 16:17:13 2022] GET /identity/v3/role_assignments?group.id=c03b057d0fae443d813860edace6d994&scope.project.id=9ac75ab2f7e84b088f06cdcdbcfae272&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 611 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:16.520209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb0515-90b5-4d96-b46b-3ba4f5256036 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:16.521194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb0515-90b5-4d96-b46b-3ba4f5256036 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:16.521605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fb0515-90b5-4d96-b46b-3ba4f5256036 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:16.523361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 223/456] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:16.529841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55b698-71e4-4039-b705-e823596780bf None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:16.530347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55b698-71e4-4039-b705-e823596780bf None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:16.530451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad55b698-71e4-4039-b705-e823596780bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:16.558215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ad55b698-71e4-4039-b705-e823596780bf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:16.559460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ad55b698-71e4-4039-b705-e823596780bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:16.637254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 234/457] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:16.713230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f30504-4e9c-401a-b06d-f6e4fea860ba None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:16.713867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f30504-4e9c-401a-b06d-f6e4fea860ba None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:16.714332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f30504-4e9c-401a-b06d-f6e4fea860ba None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:16.716171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 224/458] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:16.722108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8644ae-3956-4a45-8a28-0aec9a2422dd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:16.722650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8644ae-3956-4a45-8a28-0aec9a2422dd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:16.723152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8644ae-3956-4a45-8a28-0aec9a2422dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:16.747805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-be8644ae-3956-4a45-8a28-0aec9a2422dd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:16.749010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-be8644ae-3956-4a45-8a28-0aec9a2422dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:16.789039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:16.829678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 235/459] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:17.027239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f30504-4e9c-401a-b06d-f6e4fea860ba None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.111492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6b07a8-a7f8-45cf-bfe2-c66774bbc1f6 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.111811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6b07a8-a7f8-45cf-bfe2-c66774bbc1f6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.112048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e6b07a8-a7f8-45cf-bfe2-c66774bbc1f6 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.113673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 236/460] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:17.121176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43797ce2-9499-43f4-be00-696935652431 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.121558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43797ce2-9499-43f4-be00-696935652431 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.121806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43797ce2-9499-43f4-be00-696935652431 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.161713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-43797ce2-9499-43f4-be00-696935652431 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:17.162974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-43797ce2-9499-43f4-be00-696935652431 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:17.170613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f30504-4e9c-401a-b06d-f6e4fea860ba None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.175516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:17.229382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:17.230365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.230803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.231112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.242387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:17.244406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:17.261370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:17.262499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:17.263591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:17.263591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:17.263591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:17.265659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 225/461] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:17.270688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.298506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 237/462] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:17.299586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43797ce2-9499-43f4-be00-696935652431 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.394767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2185f047-c8f6-4919-86f0-17baaebb218b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.417017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:17.421646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43797ce2-9499-43f4-be00-696935652431 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.429888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:17.430857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.431354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.431823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.432659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:17.433183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:17.434751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbbc5aa2-a90f-42f2-a9ea-85f8c6123e08 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:17.456309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 226/463] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:17.458598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c263d87-4941-46d3-a605-00d76bc8015a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.459106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c263d87-4941-46d3-a605-00d76bc8015a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.459590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c263d87-4941-46d3-a605-00d76bc8015a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.462582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 227/464] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:17.466906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c263d87-4941-46d3-a605-00d76bc8015a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.471055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:17.471931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.472407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.472891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.484347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:17.486041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:17.488025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-11edf449-0816-4197-bff4-2de3f625da39 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:17.503940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:17.504853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:17.505689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:17.505689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:17.505689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:17.508447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 238/465] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:17.509992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.510479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.511003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.531687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:17.533620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:17.595748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c263d87-4941-46d3-a605-00d76bc8015a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.610324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 239/466] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:17.612211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.634671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:17.635642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.636097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.636542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.637376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:17.639220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:17.702810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:17.721841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaec405f-148f-45ee-b620-ab9283d05057 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.751085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 228/467] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:17:17 2022] POST /identity/v3/users => generated 248 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:17.751085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.778483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:17.779211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.779491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.779759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:17.780582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:17.780876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:17.784832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:17.811591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 240/468] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:17.816194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:17.900319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:17.947003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22969c4c-708b-454f-a77e-d28f4e3f84e9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.981101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc71c8-da1f-4f0a-a2ea-cba83e05f27f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:17.987046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:17.987683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:17.987962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:17.988234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.000541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.002203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:18.017774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:18.018816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:18.019658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:18.019658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:18.019658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:18.019658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 229/469] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:18.019658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:18.033715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:18.035015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.035481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.035903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.036720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.038324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.052035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:18.135591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 241/470] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:17:17 2022] POST /identity/v3/users => generated 246 bytes in 320 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:17:18.136915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:18.137023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12b832f-05da-4698-9d5d-5e2f82e9dec8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:18.196015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:18.224787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:18.225449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.225730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.226004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.252415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.253961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:18.280792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:18.283223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:18.284105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:18.284105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:18.284105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:18.284105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 230/471] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:18 2022] GET /identity/v3/domains/Default => generated 87 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:18.289274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:18.358824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33f39377-1acd-4b8e-8e86-a00c1f908d09 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:18.400448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:18.401246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.401525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.401791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.402441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:18.402740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.404158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1644658-9654-4e4f-a812-fdf3f4dc793b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.419662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 242/472] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:18.421728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c01b1f-a468-4e27-bf21-4f36a4664b93 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.422195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c01b1f-a468-4e27-bf21-4f36a4664b93 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.422608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c01b1f-a468-4e27-bf21-4f36a4664b93 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.424061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 243/473] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:18.425281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c01b1f-a468-4e27-bf21-4f36a4664b93 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:18.438897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50f7e442-0df3-4928-b494-5765d8ef1428 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:18.482661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:18.483230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.483517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.483784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.484425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:18.484733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.486484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f3615ea-16cd-49db-b7cf-07adfab81a16 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.493682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:18.499500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 231/474] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:18.500973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.501421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.504642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.520780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:18.522233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:18.549853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c01b1f-a468-4e27-bf21-4f36a4664b93 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:18.551333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:18.597647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:18.598787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.599130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.599398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.600051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.601474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.603052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 232/475] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:18.603910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:18.716290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 244/476] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:17:18 2022] POST /identity/v3/users => generated 249 bytes in 291 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:17:18.718742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c04a34-a1fa-4994-8a3e-769e0e339ac5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:18.757752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:18.758292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:18.758573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:18.759560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:18.760275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:18.761777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:18.873914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 233/477] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:17:18 2022] POST /identity/v3/users => generated 248 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:18.929713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:19.053121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d13844e-6989-4b99-8132-2ae5b16fb82e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:19.067300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:19.091413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:19.092387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:19.092666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:19.092939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:19.093619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:19.095069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:19.130027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.sql.core [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 27 16:17:19.130027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 27 16:17:19.130027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [parameters: {'id': '452ccac835d84636a70fb667daada132', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 27 16:17:19.130027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81612) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application conn.query(q) Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result.read() Sep 27 16:17:19.149371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return ref.to_dict() Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 16:17:19.150875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application next(self.gen) Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application yield resource Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application next(self.gen) Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.session.rollback() Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.force_reraise() Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise self.value Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application session.commit() Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 27 16:17:19.152223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._prepare_impl() Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.session.flush() Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._flush(objects) Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application compat.raise_( Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application flush_context.execute() Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rec.execute(self) Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 27 16:17:19.153096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 16:17:19.153952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application conn.query(q) Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result.read() Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 16:17:19.155395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [parameters: {'id': '452ccac835d84636a70fb667daada132', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:19.157606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Sep 27 16:17:19.159792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:19.161695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 245/478] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Sep 27 16:17:18 2022] POST /identity/v3/roles => generated 144 bytes in 222 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:19.161837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:19.187553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:19.281865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e3c0c3-c519-4a78-a40d-77eac13385bb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:19.334901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e3260d7-bed0-42e2-9b47-38f05477ee52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:19.334901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3260d7-bed0-42e2-9b47-38f05477ee52 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:19.334901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3260d7-bed0-42e2-9b47-38f05477ee52 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:19.334901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e3260d7-bed0-42e2-9b47-38f05477ee52 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:19.352077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e3260d7-bed0-42e2-9b47-38f05477ee52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1e3260d7-bed0-42e2-9b47-38f05477ee52 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:19.357199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:19.358058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:19.358878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 234/479] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:17:19 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:19.367873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:19.553387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-929f9ff4-3b6e-4d10-9bfd-226e674d6622 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:19.599485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:19.600111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:19.600387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:19.600653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:19.601320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:19.601616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:19.603669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83e2f4a-e041-4e12-b182-8cd05332aab3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:19.612320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 246/480] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:19 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:20.326445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7240dd7-b67b-4037-81e7-fcc379e2d35a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:20.326445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7240dd7-b67b-4037-81e7-fcc379e2d35a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:20.326445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7240dd7-b67b-4037-81e7-fcc379e2d35a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:20.332907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 235/481] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:20 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:20.334673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bdde92-5dfa-49a0-bb45-3d7dd10d3e92 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:20.335183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bdde92-5dfa-49a0-bb45-3d7dd10d3e92 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:20.335649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bdde92-5dfa-49a0-bb45-3d7dd10d3e92 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:20.369213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-70bdde92-5dfa-49a0-bb45-3d7dd10d3e92 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:20.370732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-70bdde92-5dfa-49a0-bb45-3d7dd10d3e92 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:20.423236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:20.455603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 247/482] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:20.641208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a7e06-a758-4591-96bb-917dca882468 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:20.641575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a7e06-a758-4591-96bb-917dca882468 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:20.641782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a0a7e06-a758-4591-96bb-917dca882468 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:20.643370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 236/483] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:20.648339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:20.648774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:20.648874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:20.681690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:20.682910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:20.768555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 248/484] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:20.837534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a7e06-a758-4591-96bb-917dca882468 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:20.937629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:20.958205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a0a7e06-a758-4591-96bb-917dca882468 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:20.992882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:20.994196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:20.994474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:20.994744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.005012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:21.010887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:21.011759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:21.012717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 237/485] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:21.017786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.171826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:21.179053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf310a65-b572-40b9-8acd-3f1fd8865aac None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:21.232278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:21.232810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.233088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.233362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.234028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:21.234531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:21.235548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.235837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:21.245058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 249/486] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:21.250527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.432819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8b3a4-304b-47d6-86ec-0ccec29e7f34 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:21.437930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb62dfdc-6b2d-4c73-8f41-1762aa0550db None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:21.501991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:21.502542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.502836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.503108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.514042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:21.514693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.514991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.515320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.528264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:21.529977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:21.530732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:21.536047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:21.541347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:21.542382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:21.543202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:21.544248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 238/487] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/roles/service => generated 85 bytes in 309 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:21.549530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:21.563004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:21.565204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:21.567306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:21.567306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:21.567306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:21.574661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 250/488] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 324 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:21.575610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.607117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:21.715290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d22e54e9-bae3-43ee-bb36-3f92e73669b2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:21.730863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb165eb-929f-4d02-a77c-f2d8b721d689 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:21.764816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:21.765707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.766090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.766370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.767263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:21.767659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:21.769007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:21.769219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db282e10-23b8-46e5-8be6-ed1300dcc40b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:21.769726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.770068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.770408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.771141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:21.771507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:21.772932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f7819d2-ec9c-4c71-b0f4-bc9787845a0d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:21.782946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 239/489] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:21.784312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc16671f-0026-4741-996d-6bc3c66e7582 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.784602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc16671f-0026-4741-996d-6bc3c66e7582 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.784872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc16671f-0026-4741-996d-6bc3c66e7582 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.786230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 240/490] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:21.786816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc16671f-0026-4741-996d-6bc3c66e7582 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.795010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 251/491] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:21.796194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.796466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.796734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.823597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:21.825429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:21.862656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:21.910805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 252/492] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:21.910805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:21.910805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:21.914485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc16671f-0026-4741-996d-6bc3c66e7582 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:21.967375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:21.968328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:21.968622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:21.968896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:21.981533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:21.983131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a8b389e6-6185-4af2-99e3-022b0a8ffd7d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:21.998052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:21.999042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:21.999896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:21.999896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:21.999896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:21.999896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 241/493] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:22.001574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d2c69b-bc87-4dd5-bcdb-38d2e1409a57 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.001949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d2c69b-bc87-4dd5-bcdb-38d2e1409a57 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.002324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d2c69b-bc87-4dd5-bcdb-38d2e1409a57 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.004043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 242/494] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:21 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:22.006115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d2c69b-bc87-4dd5-bcdb-38d2e1409a57 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.020940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2059dc4-d668-4a57-ab9b-c5eb0de818e0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.060692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.061371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.061655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.061924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.070627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.073060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-2ae514a6-f45c-454f-9ea5-e07e697c28b5 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:22.090008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:22.091025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:22.091915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 253/495] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:21 2022] GET /identity/v3/users/glance => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:22.092929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.092929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.093087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.116613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:22.117837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:22.148365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d2c69b-bc87-4dd5-bcdb-38d2e1409a57 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.166044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.188813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.189705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.190651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.191150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.191647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.192597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:22.193131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.194754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.207131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 254/496] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:22.208343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.214296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 243/497] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:22.217389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.334287 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c2effe5-6b8f-4a08-b6b7-c4aa7016305e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.339447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f85fd79-0a85-490f-8669-3b259d45231e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.375934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.376908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.377332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.377745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.379559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.380368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.380737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.381361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.382011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:22.382413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.384918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.393503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:22.399039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:22.400507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:22.401298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:22.403939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 244/498] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:22.404936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.435066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 255/499] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:22.435949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.450401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.489276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.518745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cbe30b-32ca-4e58-9627-edbf6c340254 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.539388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395a9ef1-eda3-4de0-aec4-034fc9853988 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.552302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.552930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.553211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.553479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.560649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.562104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.573835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.574424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.574704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.574996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-42d47bb6-eb7e-46e8-8b27-5fcc0d7e5d91 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:22.575185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:22.576044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:22.576932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:22.577934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:22.577934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.578069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 245/500] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:22.578134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.578198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb7c981-a8c6-4dc3-9ae5-07bbbf9d60fd None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.578198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb7c981-a8c6-4dc3-9ae5-07bbbf9d60fd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.578198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb7c981-a8c6-4dc3-9ae5-07bbbf9d60fd None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.579426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 246/501] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:22.580118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb7c981-a8c6-4dc3-9ae5-07bbbf9d60fd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.589428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 256/502] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:22.590563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.675020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.682821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb7c981-a8c6-4dc3-9ae5-07bbbf9d60fd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.689739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1298399-34c6-4395-8de5-91f9215181af None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.703102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.719415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f737c39-fae7-41a6-b119-61be1a61f2d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.720026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f737c39-fae7-41a6-b119-61be1a61f2d3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.720307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f737c39-fae7-41a6-b119-61be1a61f2d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.720577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f737c39-fae7-41a6-b119-61be1a61f2d3 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.724563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.725177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.725453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.725721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.736958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:22.742209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:22.743414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 257/503] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/roles/service => generated 85 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:22.744197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.789399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f737c39-fae7-41a6-b119-61be1a61f2d3 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.791083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f737c39-fae7-41a6-b119-61be1a61f2d3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.845891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3723ba-481e-400d-8595-c412eecac6c1 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:22.878727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:22.879347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.879624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.879890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.880601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:22.880902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:22.882340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:22.885353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.892336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 247/504] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:22.893569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:22.893852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:22.894125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:22.905439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 258/505] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:22.905439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:22.912815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:22.914027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:22.932372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:22.981817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 248/506] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:22 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:22.982649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.003235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf4a8bd5-a652-444d-bde5-be112a3edbad None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.039731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.040514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.040796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.041069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.052262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.053818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:23.069186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:23.070114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:23.072556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:23.072556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.072556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:23.074912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 259/507] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:23.076673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.097998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fbc18d-f46c-4fb9-8677-0f332afab4a2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.126698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.138070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.138614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.138922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.139200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.139856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:23.140155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.141536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.149498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 249/508] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:22 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:23.150535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.171146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.189148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2a494bd-35ae-4ea5-8186-ab4c5b80f452 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.225957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.227627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.227905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.228174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.238769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.240389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:23.256499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:23.257433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:23.262588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:23.262588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.262588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:23.262588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 260/509] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:23.262588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.262971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ebb6595-31f3-4ba1-b7eb-131cbb131a9d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.295436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.296037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.296315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.296585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.351087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.359312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2baa1365-ab2b-432e-8707-9307fab4f7ea None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.364516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.365956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.392848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.393444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.393718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.393985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.394611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:23.394944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.396289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.397640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.407935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 261/510] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:23.409347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.468227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 250/511] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:23.469161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.508732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376c8a0f-3891-4338-ac3d-6853b5da041c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.542212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.542865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.543145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.543411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.552635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.552982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.554429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.567643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e66170f5-b2b9-44c6-9700-afb72e5c7be0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:23.569177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:23.570047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:23.571638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:23.571638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.571638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:23.571638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 262/512] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:23.571638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.601581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.602138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.603038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.603316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.613093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.614616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.616515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:23.629821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:23.630800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:23.631669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:23.631669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.631669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:23.631669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 251/513] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:23.631669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.673420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0deb2a1-6284-460f-b0d3-2e402fac0d42 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.706581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.707208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.707488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.707759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.708396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:23.708699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.710076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.722169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 263/514] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:23.723094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.730458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12db75d1-2424-4b14-b30b-690d81692286 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.763923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.764555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.764831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.765097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.772326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.773765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.776058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:23.785825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:23.786766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 252/515] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/users/neutron => generated 85 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:23.794153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.822623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9acef01a-69ac-478a-8b53-d61f33529fe5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.831162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:23.856688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.857298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.857816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.858310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.869657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.871151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:23.887033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:23.887954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:23.888811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:23.888811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:23.888811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:23.888811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 264/516] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:23.889284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc3493b-64c4-4b4c-abfd-919ba45f016d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.889379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.925314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:23.925920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:23.926198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:23.926471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:23.927131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:23.927436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:23.928800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:23.940350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 253/517] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:23.941133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:23.992542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9b17df5-b23a-42af-bf75-adb7a87d9281 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:23.997735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.025229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.025795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.026073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.026341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.027007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:24.027304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.028725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.038704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17371153-8c74-4340-85bd-99e891415fd5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.040871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 265/518] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:24.041659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.059567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.076637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.077313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.077589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.077858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.084693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.086174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:24.098201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:24.099076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:24.099917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 254/519] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:24.100397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.144145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34deb5f9-ecaf-4709-8a94-901cd2d81981 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.176740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.177338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.177610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.177876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.178589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:24.178923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.180387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.197589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 266/520] 10.222.0.64 () {56 vars in 1139 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:24.198388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.200065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61eb8e78-30bf-489b-84ad-ac3e61004e9f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.217592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.233343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.233947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.234221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.234492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.235151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:24.235452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.236811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.248173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 255/521] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:24.249023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.277808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.300809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e76f95-a0b5-42cc-8479-2627ab7a4ec2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.339897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.340740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.340952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.341225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.348309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.349783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.350676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af11b59-269c-4036-b917-52e64a8440ba None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:24.361209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:24.362130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 267/522] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/users/cinder => generated 84 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:24.363123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.388305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.388910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.389187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.389456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.390092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.391544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.459929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d641e2e2-8a51-45d4-b8ea-0817c32e5f8f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.467545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 256/523] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:17:24 2022] POST /identity/v3/users => generated 247 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:24.468671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.501120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.502210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.502659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.503015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.503690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:24.503989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.509197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.511451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.526660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.529052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 268/524] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:24.530121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.604934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d7c8c5-1abe-4c9f-84a8-f7b6299fd33c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.641684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.642301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.642577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.642873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.660659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29fdea20-8126-40d0-801b-1809661252d5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.701092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.702014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.702290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.702555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.711265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.713477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.721449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.722926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:24.726709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:24.728607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 269/525] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/projects/service => generated 88 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:24.730030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.760222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.804874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:24.843992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 257/526] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:24.843992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.860858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be8d978a-53ba-49be-8e0d-dca8fe0b61c3 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.905725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.906351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.906622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.906898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.907618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:24.908037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.909469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:24.928010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 270/527] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:24.928838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:24.957672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b039e954-351b-4eba-b4a3-c54580b234c0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:24.995174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:24.995705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:24.995985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:24.996258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:24.996988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=6a27d7ca876a4320bb05f2afc86bee3d) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:24.997297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:24.998731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.000550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:25.010803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 258/528] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/role_assignments?user.id=6a27d7ca876a4320bb05f2afc86bee3d&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:25.010803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.066092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90728d14-b85b-45fe-b552-0c5716766967 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:25.089463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:25.119235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:25.119908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:25.120193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:25.120749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:25.121910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:25.122225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:25.124663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6c5ff2-a28b-4826-8e77-8c7f6c7903b8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:25.124829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.143124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 271/529] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:24 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:25.144407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.168182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:25.168985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:25.169392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:25.169768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:25.182389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:25.186592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:25.204534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:25.205497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:25.206340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:25.206340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:25.206340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:25.206340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 259/530] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:25.206765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.280517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0eee089-d088-4735-9d98-57618801c74c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:25.283716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:25.317049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:25.317800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:25.318137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:25.318468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:25.338226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-720bc0d9-78b5-461e-968f-6b7caf7d2f56 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:25.374456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:25.392175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:25.393234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:25.393940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:25.394380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:25.395533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=ab556e3087544c42929aab1b35d145a4) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:25.396018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:25.398888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.404469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:25.406087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.410858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 260/531] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/role_assignments?user.id=ab556e3087544c42929aab1b35d145a4&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:25.412327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.552415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 272/532] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:25.560734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.650175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb28447d-f19e-4759-96a9-1678e2d7ec90 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:25.673330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:25.713090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:25.715353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:25.718050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:25.718349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:25.718627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:25.719305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:25.719612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:25.721009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.750545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 261/533] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 339 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:25.755651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.764967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45211016-8d81-4bcb-add9-5ad2963c16bf None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:25.839819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:25.840375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:25.840654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:25.840923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:25.865693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:25.867815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:25.885443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:25.886372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:25.887277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:25.887277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:25.887277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:25.887277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 273/534] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 326 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:25.893710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:25.963568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e630a9c9-a379-4958-8d61-b30c708c6433 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.016881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.017638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.017929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.018205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.035996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.037683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.043017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d07036e-be04-45fd-816d-2258a4cbc283 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.075192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:26.075506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:26.076386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:26.077227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:26.077625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 262/535] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/projects/service => generated 88 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:26.084675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:26.095777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.096558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.096948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.097321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.098509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:26.098638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.100639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.129117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:26.142281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 274/536] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:25 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 249 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:26.149834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:26.243588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb518f90-b330-468a-ae2c-9466da639389 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.341141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78bc78ed-0824-465a-bad5-7509b37ec982 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.367890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.368829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.369282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.369723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.370903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:26.371453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.374071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.386667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.387332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.387606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.387880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.396598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 263/537] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 312 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:26.406812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.406812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.407050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:26.426808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:26.427690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:26.428537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 275/538] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:26 2022] GET /identity/v3/projects/service => generated 88 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:26.433305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:26.516668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:26.575146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:26.588215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c68b21-39f6-4c71-aa09-e2b12bb30ffb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.620977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27196842-9284-4875-a2e3-6776d9f32ba9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.660836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745557f-1677-468f-885d-960a01033338 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.661482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745557f-1677-468f-885d-960a01033338 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.661715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745557f-1677-468f-885d-960a01033338 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.661981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3745557f-1677-468f-885d-960a01033338 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.662820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745557f-1677-468f-885d-960a01033338 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=28f0f9fef2f54e56a7a92d077c25d2b7) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:26.663141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745557f-1677-468f-885d-960a01033338 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.664806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3745557f-1677-468f-885d-960a01033338 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.677251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.677863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.678143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.678413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.679141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:26.679455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.679757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 264/539] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:26 2022] GET /identity/v3/role_assignments?user.id=28f0f9fef2f54e56a7a92d077c25d2b7&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:26.681082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5e69c-f6f7-491e-9395-5b4ebbc382c3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.695033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 276/540] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:26 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:26.707878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745557f-1677-468f-885d-960a01033338 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:26.881147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3745557f-1677-468f-885d-960a01033338 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:26.944548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:26.945503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:26.946494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:26.946940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:26.948060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=f8babdc682094c18a4ba710224815a27) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:26.948510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:26.950723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7626e4f6-83d3-482b-a761-cf63466a217e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:26.959406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 265/541] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:26 2022] GET /identity/v3/role_assignments?user.id=f8babdc682094c18a4ba710224815a27&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:27.145202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85631271-b171-488d-951e-4dc149ba5bfe None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:27.145883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85631271-b171-488d-951e-4dc149ba5bfe None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:27.145883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85631271-b171-488d-951e-4dc149ba5bfe None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:27.147024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 277/542] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:27.163683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:27.164005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:27.164234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:27.197297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:27.198586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:27.226399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:27.295064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 266/543] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:27.641135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85631271-b171-488d-951e-4dc149ba5bfe None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:27.762172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85631271-b171-488d-951e-4dc149ba5bfe None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:27.770683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:27.810408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be0ef664-6803-4970-8db0-442378c8e628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:27.811188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ef664-6803-4970-8db0-442378c8e628 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:27.811476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ef664-6803-4970-8db0-442378c8e628 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:27.811792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be0ef664-6803-4970-8db0-442378c8e628 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:27.829107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be0ef664-6803-4970-8db0-442378c8e628 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-be0ef664-6803-4970-8db0-442378c8e628 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:27.834108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:27.835024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:27.835894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 278/544] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:27 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:27.841411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:27.960641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc23f707-84e5-4ff0-bfa3-6299d07b7350 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:27.968922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc52a0b-b2bc-4b52-a360-7d607cae032b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:27.969282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc52a0b-b2bc-4b52-a360-7d607cae032b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:27.969518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc52a0b-b2bc-4b52-a360-7d607cae032b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:27.971124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 279/545] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:27.978668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e0192-f8ae-4894-9b50-4e56a57efde8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:27.979007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e0192-f8ae-4894-9b50-4e56a57efde8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:27.979241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194e0192-f8ae-4894-9b50-4e56a57efde8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.002304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.002952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.003236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.003512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.004183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:28.004490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:28.006034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:28.015837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 267/546] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:27 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:28.021241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-194e0192-f8ae-4894-9b50-4e56a57efde8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:28.022481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-194e0192-f8ae-4894-9b50-4e56a57efde8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:28.027930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.047339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:28.145061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32d4ef6f-7d1e-4f0a-bfe0-5c44870ec48a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:28.165914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 280/547] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:27 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:28.167509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19bd023-f3fb-4ba6-9993-55d7553aee34 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.167805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19bd023-f3fb-4ba6-9993-55d7553aee34 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.168077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d19bd023-f3fb-4ba6-9993-55d7553aee34 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.170056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 281/548] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:28.176306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.176587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.176882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.191393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.192187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.192588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.192954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.202586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:28.204524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:28.219934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:28.221348 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:28.249028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:28.251648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:28.254163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:28.254163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:28.254163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:28.254163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 268/549] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/domains/Default => generated 87 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:28.263802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.293332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 282/550] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:28 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:28.382149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:28.434624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68690311-30d9-4aed-bab7-49f96d86764e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:28.489445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.490216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.490329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.490726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.491093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.492045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:28.492491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:28.495159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:28.518864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 269/551] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:28.525519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.531268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:28.620179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e34adb7-986a-4373-9e59-f0452535b83c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:28.656175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.656815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.657094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.657365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.668882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:28.673889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:28.674830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:28.675662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 283/552] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:28.681625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.732226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45584ed7-d906-44ee-8a8f-f9d30a1582e6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:28.762272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:28.772346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.774642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.774642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.774642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.780792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:28.783078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:28.784795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:28.797047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe6e002b-8f4c-4da3-a590-4a2494c905b6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:28.798926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:28.800054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:28.801155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:28.803281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 270/553] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/users/swift => generated 83 bytes in 278 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:28.804308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.840551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.841197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.841464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.841738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.859326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:28.864895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:28.865984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:28.866976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:28.874828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 284/554] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/roles/service => generated 85 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:28.874828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:28.947926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b40ead7-f4ce-4939-993f-5b4344f82bf8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:28.992111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:28.993667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:28.993950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:28.994222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:28.994933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:28.995239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:28.997105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.008959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 271/555] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:29.009924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.010651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74ee7a-b30d-4647-bbe2-9fd533428f49 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.048069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.049145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.049916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.050634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.051589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:29.052091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.053726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.072338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.074705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 285/556] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Sep 27 16:17:28 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:29.075899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.080034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.139095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-518bd10d-b68e-49d3-94af-800cf7bd322e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.184048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.185128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.185417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.185692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.186422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:29.186934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.188572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.201283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdc7b12a-9d9e-4ec0-8119-5b52c0a87f14 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.201819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 272/557] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:29.202163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.241974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.244404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.244905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.245764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.256630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.258982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:29.276761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:29.277982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:29.278891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:29.278891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:29.278891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:29.285323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 286/558] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:29.285323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.342259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71b9ddd6-9c80-4566-a073-65e9cbfadad1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.357673 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.372497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.391762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03a4992f-c46a-4c3d-be02-256c4b529fb9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.392384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a4992f-c46a-4c3d-be02-256c4b529fb9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.392662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a4992f-c46a-4c3d-be02-256c4b529fb9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.392932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03a4992f-c46a-4c3d-be02-256c4b529fb9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.428504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3066674-fb91-4b94-a408-3484f7f2d82d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.468649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.469728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.470251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.470851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.481197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a4992f-c46a-4c3d-be02-256c4b529fb9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.483085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03a4992f-c46a-4c3d-be02-256c4b529fb9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.483795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.485237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:29.500151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:29.501111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:29.501986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:29.501986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:29.501986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:29.501986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 287/559] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:29.502206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.616721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5577953-a442-4332-aa1f-71dc877b7e52 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.620288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 273/560] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:29.624154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5bb08c-6917-4721-9688-134f44591446 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.624515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5bb08c-6917-4721-9688-134f44591446 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.625525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5bb08c-6917-4721-9688-134f44591446 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.628050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 274/561] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:29 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:29.628050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5bb08c-6917-4721-9688-134f44591446 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.631888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.642111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.653611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.654214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.654492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.654763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.655431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:29.655740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.657111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.677086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 288/562] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:29.677086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.744279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5bb08c-6917-4721-9688-134f44591446 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.780521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-360620d9-de72-40ce-892c-6757f395dd48 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:29.783667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.784278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.784556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.784825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.785461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:29.785761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.787153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-775d0a49-35ab-470c-b4f9-7a1700edce4b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.799977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 275/563] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:29.801268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.801630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.801908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.820302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:29.821894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:29.826139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:29.827140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:29.827599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:29.828338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:29.842548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:29.844471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:29.861318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:29.862324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:29.863225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:29.863225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:29.863225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:29.865011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 289/564] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:29.865937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:29.892661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.894962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:29.897460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 276/565] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:29.903686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.002593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912fa394-37cf-473f-918a-5878dc8abc28 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.011733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a53559e-6cd6-4b5a-9b81-af5bce492d56 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.038843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.039562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.040315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.040590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] PATH_INFO: `/v3/users/glance` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.048092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.048913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.049522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.049864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.050293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.050699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.058861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.060514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:30.060650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:30.061689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 27 16:17:30.062847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:30.066369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 290/566] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/users/glance => generated 84 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:30.067211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0fc732a4-4b24-4fe4-be24-c08cface5e48 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:30.076352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:30.077297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:30.078507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:30.079077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 277/567] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:17:29 2022] GET /identity/v3/users/nova => generated 82 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:30.080105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e64dc4-d102-479e-aff8-444191f67033 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.080385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e64dc4-d102-479e-aff8-444191f67033 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.080655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e64dc4-d102-479e-aff8-444191f67033 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.085915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 278/568] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:30 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:30.085915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e64dc4-d102-479e-aff8-444191f67033 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.151522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.158369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.176884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e31cd8b8-678b-4f07-8b4f-7f80002c24f7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.193254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e64dc4-d102-479e-aff8-444191f67033 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.216770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.217724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.218188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.218632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.219753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:30.220297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.221931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.234039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 291/569] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:30.234962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.236676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.237287 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.237620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.237891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.238638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:30.239039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.240477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43a17a89-7a96-4232-ac4c-54f2a465415f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.277605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 279/570] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:30.278856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.279140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.279409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.296941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:30.298124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:30.346082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595df687-52cc-4cc5-bd54-c5b64200fd2f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.365511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 280/571] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:30.366341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.382832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.383962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.384329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.384599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.385324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:30.385627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.387063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.395812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.419799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 292/572] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:30.419799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.426329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.479292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04897aec-64a6-40dd-9ceb-09c87f17eadf None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.514592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.515304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.515580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.515846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.516691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f791a04-d33c-40f7-bd0d-7844d030c07f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.526785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:30.532146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:30.533134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 281/573] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:30.533988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.559722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.560352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.560628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.560895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.567562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.568991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:30.580443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:30.581337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 293/574] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:30.583076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.645104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.649672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7e8c34-bc66-4753-80e9-40f08f26597a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.658961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.684774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.685334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.685606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.685878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.688676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c3d5092-4851-4fd0-9373-faaa4b6a89a7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.695610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.697071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:30.713733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:30.714928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:30.716203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:30.716203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:30.716203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:30.716203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 282/575] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:30.716203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.724278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.725032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.725406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.725798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.736669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.738092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:30.753315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:30.754404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:30.755506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:30.755506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:30.755506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:30.755506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 294/576] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:30.755794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.816115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f00f0d4-4bfc-43a7-8f65-1ccc3e5e87e0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.851084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.851680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.852588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.852934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.853503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:30.853800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.854222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37cd1ab-a6dd-434d-98bf-2acea162e415 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.855192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.863563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 283/577] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:30.864420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.864650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.887604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.888206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.888486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.888757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:30.889449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:30.889716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:30.889825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:30.891211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:30.903862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 295/578] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:30.906375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:30.964240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3899acfa-a22d-4138-9423-debd51750cb3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:30.997787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:30.998438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:30.998714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:30.999016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.002620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fbbe3fb-9525-4432-853c-4f34f8bc6589 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.008816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:31.013631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:31.015176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 284/579] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:31.016207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.036946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.037552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.037829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.038094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.038723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.039054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.041014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.051938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 296/580] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.052879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.083326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.097299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.114990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc7a2b08-3a5f-44f2-bedc-eeec73676214 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.148675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.149288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.149551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.149823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.150451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.150754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.151705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4d800b-a9d3-4077-a85b-13dfb8e502a6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.152146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.163669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 285/581] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.165184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.186541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.187193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.187470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.187741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.198271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.199758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:31.214732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:31.215831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:31.216720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:31.216720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:31.216720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:31.216720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 297/582] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:31.216720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.265980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c32e426-7a46-4a48-aa96-bdbfc6ee4bea None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.299652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.300242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.300522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.300790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.301489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=85bbaf87748242bf830ca84267692c82) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.301790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.302753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.303253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.308530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 286/583] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/role_assignments?user.id=85bbaf87748242bf830ca84267692c82&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.309355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.319039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.322361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c645a207-48b0-412b-ba52-097ea1aaf43b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.361290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.361915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.362193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.362463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.363189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.363888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.364897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.375923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 298/584] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.376892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.442577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e343500-0612-4d3e-8f50-c7fd34c3e51c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.480766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.481332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.481610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.481882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.491118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147ee8f0-09ea-4e09-ba1e-4849041a41fc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.492307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.494272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:31.506084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:31.507248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:31.508875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 287/585] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:31.509367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.534747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.536110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.536541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.537205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.545687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.547320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:31.561879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:31.562912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:31.563806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:31.564232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 299/586] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:31.565024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.569004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.586963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.623762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f404361-d212-4bee-a431-956b9b79ab40 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.659248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.659951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.660236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.660507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.661150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.661458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.662851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.674300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddeeb659-fe7e-4d55-9135-f9184085d30a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.675822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 288/587] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.676634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.714281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.714935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.715216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.715493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.726441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.728264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:31.743695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:31.744642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:31.745521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:31.745521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:31.745521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:31.745521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 300/588] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:31.745521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.791823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6058010-214d-459a-bdc2-d52d3a6a2c5b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.809796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.818858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:31.829281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.829895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.830175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.830445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.831166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.831473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.832910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.853272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 289/589] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.854297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.861292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a548ae5e-027d-4f45-b7a7-c8fc873bbaf2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:31.896536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:31.897373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:31.897821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:31.898233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:31.899347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:31.899761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:31.901213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:31.915349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 301/590] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:31.916408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:31.968276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7006bfa-ffaa-42bd-9aba-79e249e1c2ce None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.003974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.004632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.004904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.005170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.013749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.015217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.016441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d9c346-ccfe-4a9e-b770-ac0265ca2557 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:32.027371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:32.028335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 290/591] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/users/neutron => generated 85 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:32.029323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.040392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:32.060041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.060762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.061089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.061478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.062110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:32.062403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.063902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.077592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:32.082992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 302/592] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:32.082992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.140763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26877c6f-b024-42de-a084-a2f35a96da43 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.177110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.177765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.178040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.178331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/6a27d7ca876a4320bb05f2afc86bee3d/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.190959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ab2150-3035-4c7a-91a7-05aa2752099a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.192987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=6a27d7ca876a4320bb05f2afc86bee3d, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.194427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.227966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.228724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.228995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.229269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/ab556e3087544c42929aab1b35d145a4/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.244476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=ab556e3087544c42929aab1b35d145a4, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.245979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.298038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 291/593] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:32 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/6a27d7ca876a4320bb05f2afc86bee3d/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:32.298824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.407330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 303/594] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:32 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/ab556e3087544c42929aab1b35d145a4/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 322 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:32.409275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.432583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:32.442290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a570d9-d395-4e51-8421-3bc819eca64f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.478649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:32.516377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.517423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.517757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.518159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.519254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:32.519563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.521012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.548802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 292/595] 10.222.0.64 () {56 vars in 1139 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:32.554995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.584925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57a20aa-0c2f-4ec3-938a-573be3c79087 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.646024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.647354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.647642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.648062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.658451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.660609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:32.677799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:32.679201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:32.680811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 304/596] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/users/cinder => generated 84 bytes in 271 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:32.685744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.714381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83436bce-b55c-48ab-82f6-34353f51478c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.752151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.752778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.753049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.753310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.764239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.765679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:32.782065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:32.784336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:32.786404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:32.786404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:32.786404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:32.786404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 293/597] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 229 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:32.794801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.799758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:32.807491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:32.827371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca0a17c4-715e-41e4-91ed-8033001560e9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.881680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.882385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.883854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.884311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.885915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:32.886452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.889138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.919183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1464bcfc-5871-491d-939c-5b377c2c3f20 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:32.926934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 305/598] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:32.937305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:32.958170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:32.959258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:32.959710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:32.960154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:32.960979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:32.961989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:32.963773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:32.977474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 294/599] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:32.983106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.072391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa017978-5f2f-46d5-88ea-89b96647cc81 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.119191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:33.127234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.127884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.128160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.128428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.145881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.149272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.163531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:33.179247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:33.180589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:33.181581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:33.181581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:33.181581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:33.182626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8209eb-b618-46dc-a8d3-8599bf5841b9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.185966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 306/600] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:33.189347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.252798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.253438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.253715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.253986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.268358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.269828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:33.285642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:33.286563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:33.287470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:33.288358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 295/601] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:33.295499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.386434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9bf264-4d9a-4096-aedd-31bf65ed2dcd None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.453305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.453919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.454196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.454474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.455149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:33.455449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.456820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.482662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 307/602] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:33.489907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc2dbbf7-0c15-4d86-b6c2-9420565e8850 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.492104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.536323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:33.551608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.552224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.552501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.552770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.556129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:33.556448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.557897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.584167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 296/603] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:33.592215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.605705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:33.682680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca3066e0-faa4-4edc-9352-f5c7b7eaab25 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.749980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.750496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.750808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.752423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.759037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.760922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:33.773149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:33.774195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:33.775108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 308/604] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:33.782709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.788436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d749dd47-eb81-4f7a-983e-929285db73d3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.838882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.840030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.840306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.840579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/28f0f9fef2f54e56a7a92d077c25d2b7/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.860859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=28f0f9fef2f54e56a7a92d077c25d2b7, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.862455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.906680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3b61902-8169-4537-a9fe-2d65377156ea None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:33.922876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:33.939859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:33.940465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.940743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.941004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.941689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:33.942011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:33.943484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-036cf02e-4c91-431f-b52f-8b796a7a3917 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:33.957778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 309/606] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:33 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:33.958114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 297/606] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:33 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/28f0f9fef2f54e56a7a92d077c25d2b7/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:17:33.959339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e63fd-79e9-4607-b4e7-95c8f7d43684 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.959440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e63fd-79e9-4607-b4e7-95c8f7d43684 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.959681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475e63fd-79e9-4607-b4e7-95c8f7d43684 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.960897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 310/607] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:33.965015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:33.967362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:33.967644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:33.967917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:33.992829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:33.994584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:34.014398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:34.076931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 311/608] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:34.134149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a257c4e-e32b-47fc-bb4e-e3393528f9ea None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:34.178338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:34.179049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:34.179330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:34.179604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/f8babdc682094c18a4ba710224815a27/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:34.205204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=f8babdc682094c18a4ba710224815a27, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:34.207075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:34.282126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 298/609] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:33 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/f8babdc682094c18a4ba710224815a27/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 318 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:34.533537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:34.645288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04d64409-2daf-4941-8401-e38db1b8f91e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:34.653864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:34.683121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a293d7db-ec43-4dfc-8569-575e9ad15f4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:34.684093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a293d7db-ec43-4dfc-8569-575e9ad15f4a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:34.684597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a293d7db-ec43-4dfc-8569-575e9ad15f4a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:34.685242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a293d7db-ec43-4dfc-8569-575e9ad15f4a None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:34.697065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a293d7db-ec43-4dfc-8569-575e9ad15f4a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a293d7db-ec43-4dfc-8569-575e9ad15f4a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:34.702251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:34.703537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:34.704562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:34.706867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 312/610] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:34 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:34.712672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:34.856212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:34.876691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae00a1b-7fea-4363-a020-e8394ecbfc22 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:34.900905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a378b-1dcf-4d5e-9f1a-2e5e867e3f93 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:34.901208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a378b-1dcf-4d5e-9f1a-2e5e867e3f93 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:34.901456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90a378b-1dcf-4d5e-9f1a-2e5e867e3f93 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:34.903093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 313/611] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:34.908193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2a766-c692-47fe-b0b8-2eb138ad90d1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:34.908471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2a766-c692-47fe-b0b8-2eb138ad90d1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:34.908745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2a766-c692-47fe-b0b8-2eb138ad90d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:34.929839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:34.930453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:34.930734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:34.931037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:34.931701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:34.932002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:34.933371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:34.935502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f5e2a766-c692-47fe-b0b8-2eb138ad90d1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:34.936705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f5e2a766-c692-47fe-b0b8-2eb138ad90d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:34.951981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 299/612] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:34 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:34.957594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.088824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 314/613] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:35.094122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a62d584-003d-48cb-be32-0cc9b5c6b6ab None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.108772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043db678-39cf-4ec9-a3b1-894f5633d6dd None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.109340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043db678-39cf-4ec9-a3b1-894f5633d6dd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.109834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-043db678-39cf-4ec9-a3b1-894f5633d6dd None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.112015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 315/614] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:35.117969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.118536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.119034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.138921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:35.140436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:35.170307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.171635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.172115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.172694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.188120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:35.189966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:35.205950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:35.206960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:35.207869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:35.207869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:35.207869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:35.209862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:35.210293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 300/615] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:35.223682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.245318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 316/616] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:35.369852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03e2bf38-5c96-4471-92f0-e1b04a0761b7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.406068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.407081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.407575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.408104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.408959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:35.409489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:35.411129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:35.424416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 301/617] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:35.431934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.469016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:35.545279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcc3b7d1-f31d-42ce-8404-f7ea6635895a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.581317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.582314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.582877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.584330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.590023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.598083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:35.599678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:35.612694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:35.613767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:35.614832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 317/618] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/users/swift => generated 83 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:35.620098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.653894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:35.702478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeb59d0d-12bc-47a6-a944-499cff533d21 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.713328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:35.730246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-782bff96-59ed-4d91-abbb-ed2d1fad0574 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.740742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.741684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.742338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.742890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.754950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:35.761296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:35.762231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:35.763289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:35.765778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 302/619] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:35.766714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.778602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.778602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.778602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.778602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.789761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:35.796463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:35.797525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:35.798358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:35.800874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 318/620] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/roles/service => generated 85 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:35.801828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.905246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ff9cc6-7f1e-41d0-a29b-7ed24209f100 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.923531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aea721c6-bbac-40bb-a7f5-858e43036197 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:35.934235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:35.955694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.956926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.957359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.957792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.958662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:35.959241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:35.960761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:35.961686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:35.962281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:35.962660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:35.963069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:35.963833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:35.964240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:35.966025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:35.974562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 319/621] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:35.975268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.985230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 303/622] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 218 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:35.987055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:35.988242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.089574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804303bb-03f3-4750-b9cf-28510846a12f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.099060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182b37bd-3ba4-4a1c-844c-68d611932e77 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.132703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.134576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.135121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.135560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.136371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:36.136918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.137365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.138098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.138477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.138878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.139133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.153750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 320/623] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:36.154542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.154673 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.155876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:36.176626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:36.178038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:36.178970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:36.178970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:36.178970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:36.178970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 304/624] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:36.179666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.218730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.235201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.263081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25366836-1f28-4c2e-b75f-1a3eb3f16d8a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.309120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a330431a-29fa-4a25-969d-bdcd313bf8d4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.310205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.310876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.311159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.311430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.326671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.328207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:36.345997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:36.346976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:36.353147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:36.353147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:36.353147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:36.353147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 321/625] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:36.353147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.369504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.370540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.371092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.371583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.382735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.385200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:36.401546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:36.402556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:36.403858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:36.403858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:36.403858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:36.405372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 305/626] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:36.406431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.466670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-045163f3-5bd3-40c0-813c-054de94fb645 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.493916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.497845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.505501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7678528-6dea-4649-a965-caee044a897d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.506109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7678528-6dea-4649-a965-caee044a897d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.506390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7678528-6dea-4649-a965-caee044a897d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.506659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7678528-6dea-4649-a965-caee044a897d None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.507347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7678528-6dea-4649-a965-caee044a897d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:36.507665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7678528-6dea-4649-a965-caee044a897d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.509871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7678528-6dea-4649-a965-caee044a897d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.516829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d2b17a6-8a63-4f45-98be-e9d843c135cd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.531132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 322/627] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:36.531132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4592dbf0-1203-4cec-8977-63a610a61475 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.531132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4592dbf0-1203-4cec-8977-63a610a61475 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.531132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4592dbf0-1203-4cec-8977-63a610a61475 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.531132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 323/628] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:36.531132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4592dbf0-1203-4cec-8977-63a610a61475 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.553528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.554233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.554509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.555308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.556057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:36.556361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.557881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.571002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 306/629] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:36.572417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.643690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4592dbf0-1203-4cec-8977-63a610a61475 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.678746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179584e8-f7b2-41cf-8254-79650e956273 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.685330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.686020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.686216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.686467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.687140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:36.687440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.688800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e151ada-df91-4099-aee7-a566229ecc8b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.703844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 324/630] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:36.705814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.706281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.706733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.718808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.719444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.719726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.720001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.728039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:36.728997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.729228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:36.730432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:36.741727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:36.743303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 307/631] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/users/nova => generated 82 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:36.744275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.747618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.774855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:36.809406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 325/632] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:36.810257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.870138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2611b547-b021-4e12-a5e6-5d7d952440d8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.923173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.923819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.924102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.924378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.932316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92c31d91-15bd-4a1c-9af1-d896157ba37a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:36.934238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.936894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:36.953690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:36.954647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:36.955623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:36.958913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 308/633] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/projects/service => generated 88 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:36.960015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:36.976496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:36.977295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:36.977574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:36.977792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] PATH_INFO: `/v3/users/glance` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:36.986557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:36.988049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:37.001612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:37.002851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:37.004018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 326/634] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/users/glance => generated 84 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:37.008893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.041711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.045757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.082225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2571d19d-d099-4cb9-9f61-4f62bf31bf11 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.117247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.117857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.117986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efa358c8-d64d-4da1-9245-06470785742c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.118157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.118408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.119160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:37.119465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.120892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b32647-d4ba-4b7a-af96-9f656dc3c02b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.138571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 309/635] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:17:36 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:37.139894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0627923-bb6e-4d90-8e36-385578dfce82 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.140240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0627923-bb6e-4d90-8e36-385578dfce82 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.140522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0627923-bb6e-4d90-8e36-385578dfce82 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.141711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 310/636] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:37.142466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0627923-bb6e-4d90-8e36-385578dfce82 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.156947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.157554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.157833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.158101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.158805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:37.159116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.160540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.173023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 327/637] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:37.173648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.244595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0627923-bb6e-4d90-8e36-385578dfce82 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.263777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.264925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.272108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb784ab7-8549-4536-9a46-f2696371eacd None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.278014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.278620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.278937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.279211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.279925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:37.280229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.281639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.299154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 311/638] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:37.300144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.312796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25f1f9f-09ca-46cb-ae8c-08898ce6e1d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.313418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f1f9f-09ca-46cb-ae8c-08898ce6e1d7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.313690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f1f9f-09ca-46cb-ae8c-08898ce6e1d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.314012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25f1f9f-09ca-46cb-ae8c-08898ce6e1d7 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.326713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25f1f9f-09ca-46cb-ae8c-08898ce6e1d7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d25f1f9f-09ca-46cb-ae8c-08898ce6e1d7 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:37.331798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:37.332797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:37.333982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 328/639] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:37.338841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.339125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.339396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.356922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:37.358132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:37.415420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56ebbbc4-68bf-4799-80da-e0c57f93a98a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.422635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 329/640] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:37 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:37.427634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.450494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.451322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.451602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.451874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.501436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.513938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.525193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.526643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.532256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e3782b-24c8-4803-8755-f44d01495404 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.568075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.569236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.569823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.570576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/85bbaf87748242bf830ca84267692c82/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.586935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=85bbaf87748242bf830ca84267692c82, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.588460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.627868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 312/641] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:37.628970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.650415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 330/642] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:37 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/85bbaf87748242bf830ca84267692c82/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:37.659307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.738111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.762061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745723a9-212d-45b9-98a6-d514d1fd286e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.771149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f20452-8d0a-4803-8e20-58ede063eb28 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.800672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.801357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.801647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.801920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.807309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.812688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.813307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.813581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.813850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.814514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:37.814908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.816331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.826473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 331/643] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:37.827405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:37.890432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.892307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.944736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-397aeb6d-d5a6-4793-be3b-234253075ea9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:37.976365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:37.983948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:37.988347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:37.988347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:37.988347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:37.996952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:37.998444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:37.999306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 313/644] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:37.999989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:38.021252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.022409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:38.023980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:38.023980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.023980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:38.023980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 332/645] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.023980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.072861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.109031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d772222-0ab5-431e-b08e-c0317779ef6a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.129497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71c6808-ed4a-4845-8a76-e960db10ac19 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.143961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.144595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.144875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.145149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.156916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:38.161763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.162671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 314/646] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:37 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.163664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.173106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.173739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.174428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.174706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.186456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.187936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:38.203515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.204477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:38.205374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:38.205374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.205374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:38.206743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.207768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 333/647] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.208560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.272734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086cd96d-b67b-4117-877d-8b130f487e02 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.306884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.307615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.308006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.308277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.310481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.314260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae27f73-5504-4d77-a7d3-e85fc8502974 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.319546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.321785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:38.341547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.342698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:38.343784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:38.343784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.343784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:38.349687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 315/648] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.349687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.358879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.359523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.360131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.360413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.361221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:38.361530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.363509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.376388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 334/649] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:38.378109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.443749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.455067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f993e6d0-aba5-46fd-a1dc-10f2ab99f794 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.483701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34ee386e-e3fe-4314-8226-f5391922b6f0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.490955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.491582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.491858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.492753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.493422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:38.493709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.495102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.508401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 316/650] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:38.508401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.519743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.520402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.520730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.520996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.521627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:38.521969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.523502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.536021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 335/651] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:38.536844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.560895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.627981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72b644b-e737-4866-9bfc-c8eeec5bcf37 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.647504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6c25cd2-f0bd-438a-980f-64f05ccbe698 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.665077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.665694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.666066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.666338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.667000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:38.667301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.668762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.683517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.684144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.684417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.684683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.685540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 317/652] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:38.686447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.691620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.693054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.702301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:38.705408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.706268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:38.707214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:38.711470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 336/653] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.712730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.794958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.800858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126b0890-9178-4cfc-9f15-5d3c29409629 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.815979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f5ada2e-343f-4b76-96f1-ea817bbe6747 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.840005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.840790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.841064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.841649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.851993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:38.852637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:38.852994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:38.853258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:38.854960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.856396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.862154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:38.863720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:38.870982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.871914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:38.872802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:38.872802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:38.872802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:38.875166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 318/654] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:38.875273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:38.876695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Sep 27 16:17:38.878369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:38.879350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.880697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 337/655] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/users/neutron => generated 85 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:38.881420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:38.945956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:38.985275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9673522-45af-4bbc-bd6c-666587cc0033 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:38.987913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05c8ef23-581c-408e-9221-ae51765e6e40 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.019593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.020212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.020486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.020751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.023176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.024746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.025032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.025307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.026025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.026333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.027787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.028330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.029758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.035442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:39.041764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:39.049646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:39.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:39.051309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 338/656] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.051309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.051452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 319/657] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:38 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:39.051452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.150267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95783b0-5a66-4716-a0fc-1d6d8a605a7f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.154138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3faae7-c116-4ad6-aec3-ae3800349166 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.178246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.184366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.187509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.187509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.187509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.187509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.187509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.188078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.190422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.191089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.191381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.191649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.192422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.192728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.194262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.204062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 339/658] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.204062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.221764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 320/659] 10.222.0.64 () {56 vars in 1139 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.221764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.268257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.312530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bd2ccd6-e3ba-4697-b448-fe51b172a0fe None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.327909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cadac01-2166-4969-b14e-1850907fd49a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.356450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.357151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.357497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.357847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.358643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=ab556e3087544c42929aab1b35d145a4) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.359047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.360537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.363082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.363934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.364052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.364236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.365019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.365335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.366748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.379191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 340/660] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/role_assignments?user.id=ab556e3087544c42929aab1b35d145a4&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.379191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.392515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 321/661] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.393499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.443532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.502128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdeebbea-d92b-4392-85c3-9bba41b8b005 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.512239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-357e4f16-4f99-4386-95cc-7ade24d1e48d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.541314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.546913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.547919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.548439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.549107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.550257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.550930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.551210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.551479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.559655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.561530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:39.576215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:39.577715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:17:39.578856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:39.581662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 341/662] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/users/cinder => generated 84 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:39.582644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.631856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.634111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.678014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.703328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbfc1891-da1b-4c40-9d8c-85bc0ec69f35 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.747708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 322/663] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:39.748115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.748657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.748971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.749322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.749668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.750512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=6a27d7ca876a4320bb05f2afc86bee3d) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.750934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.754028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.762468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 342/664] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/role_assignments?user.id=6a27d7ca876a4320bb05f2afc86bee3d&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.763904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:39.860138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.906867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59a645ad-ff95-49e5-9c31-204cec2c2074 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.930407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e72e4a1-83a2-4a2a-9a5d-8451f1324767 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:39.959284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.960490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.961012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.961764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.962831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:39.963284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.965729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a5a3fb0-2308-4cea-9fab-5663eb6fec97 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.977319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:39.978120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.978498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.978887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.984384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:39.989487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 323/665] 10.222.0.64 () {56 vars in 1137 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 241 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:39.994959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:39.995947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556c3bf-05e3-4dc1-9391-6baf00777261 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:39.996234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556c3bf-05e3-4dc1-9391-6baf00777261 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:39.996389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:39.999795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0556c3bf-05e3-4dc1-9391-6baf00777261 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:39.999795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 324/666] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:39 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:40.000588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556c3bf-05e3-4dc1-9391-6baf00777261 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-66ef1232-e957-4633-aec1-cef23d278aed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:40.013767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:40.014908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:40.015896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:40.015896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:40.015896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:40.019150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 343/667] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:40.020494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.020775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.021051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.042204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:40.043427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:40.122595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 344/668] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:40.123466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.142845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0556c3bf-05e3-4dc1-9391-6baf00777261 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:40.185360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:40.207771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:40.209313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.209585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.209851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.299474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:40.303707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:40.305695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:40.322867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88775578-905c-45d4-a4de-0c03ef0c98de None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:40.389106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:40.389861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.390137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.390404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.391074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:40.391373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:40.392733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:40.416992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 345/669] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 294 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:40.421036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.481119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 325/670] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:39 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:40.482296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.528651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:40.545960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-312daf49-ee6e-4609-af1a-e965f34c05c0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:40.592475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:40.593776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.594064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.594465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.596948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:40.600015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a37cda18-2f6f-4f54-b9e6-ea129c307919 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:40.605445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:40.607073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:40.621139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:40.622197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:40.623208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:40.629026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 346/671] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:40.629026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.651261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:40.651643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.651925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.652198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.663079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:40.670652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:40.671605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:40.672957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:40.676781 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 326/672] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/roles/service => generated 85 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:40.678080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.761118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e9fd0ef-aba6-4673-bda9-e705883da086 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:40.797923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:40.798550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.798861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.799139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.806885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea1d4e1a-907c-430a-aa62-796b7bb7d238 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:40.810868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:40.812176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:40.825145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:40.834139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:40.837820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:40.838906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:40.838906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:40.838906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:40.839848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 347/673] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:40.841007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.861530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:40.862322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:40.862703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:40.863105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:40.863905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:40.864319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:40.865863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:40.879806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 327/674] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:40.880777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:40.913191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:40.968271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18cce8df-f40c-4f1a-a32f-8864675e05c9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.011119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.011887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.012430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.012813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.013596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:41.014162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.016334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.017847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02a7415a-4a6c-4d4a-9c7e-0a16db92ec73 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.031017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 348/675] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:41.032033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.055704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.056369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.056649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.056978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.057614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:41.057920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.059409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.071694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 328/676] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:41.072541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.107045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:41.160180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:41.166709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481210ce-e697-411e-956b-1afdd73a2006 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.179840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c183d0d1-f0e2-4805-802c-875ec244449f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.209164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.210266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.211373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.212092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.213595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=28f0f9fef2f54e56a7a92d077c25d2b7) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:41.214508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.216748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.218980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.219788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.220071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.220339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.230807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 349/677] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/role_assignments?user.id=28f0f9fef2f54e56a7a92d077c25d2b7&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:41.230807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.247946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.249424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:41.279980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:41.281027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:41.281924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:41.281924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:41.281924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:41.282953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 329/678] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:41.288698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.447126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f3cd521-370c-4626-b58c-6790bf53ab04 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.479336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:41.488119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd1af91f-80c0-47ff-bd0b-bd596408a7c0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.490194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.491143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.491637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.492185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.501024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.503180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:41.522442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:41.525262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:41.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:41.527686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:41.529809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 350/679] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:41.533124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.533747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.534146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.534390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.535067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:41.535418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.535835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.536959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.557854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 330/680] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:41.558848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.682497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bfc0120-79a6-4881-8917-3232e42facf9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.697404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc294ad-579e-4191-a062-5f700ccf79a2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.737020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.738005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.738510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.739038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] PATH_INFO: `/v3/users/swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.748710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.750568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:41.765127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:41.766214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:41.767198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:41.769950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 331/681] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/users/swift => generated 83 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:41.772538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.773758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.774296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.775339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.778440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.779414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:41.780337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.782446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:41.811486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 351/682] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 276 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:41.812816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:41.866403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:41.918580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:41.942089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3e1a58-ac1f-4775-b663-291ed7b7dc2f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.991098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:41.992971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:41.993487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:41.993960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:41.994315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-401b390c-bf70-4e84-8aa5-e3cba3c8246c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:41.995134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:41.995893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:41.998571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f16e5d4-aa41-437b-b006-fcdc05faf405 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:42.022851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 332/683] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:42.024475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6fe9a4-72a8-49cf-8ecf-77a585a97198 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.024962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6fe9a4-72a8-49cf-8ecf-77a585a97198 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.025455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6fe9a4-72a8-49cf-8ecf-77a585a97198 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.026895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 333/684] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:42.028226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6fe9a4-72a8-49cf-8ecf-77a585a97198 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:42.035791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:42.036396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.036671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.036935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.037652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=f8babdc682094c18a4ba710224815a27) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:42.037962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:42.039841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61f17eea-034d-41a5-997d-060528f6d98f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:42.050808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 352/685] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:41 2022] GET /identity/v3/role_assignments?user.id=f8babdc682094c18a4ba710224815a27&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:42.050808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729f544-ca93-4868-98a0-fec7a28147a8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.050808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729f544-ca93-4868-98a0-fec7a28147a8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.050808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e729f544-ca93-4868-98a0-fec7a28147a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.084791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e729f544-ca93-4868-98a0-fec7a28147a8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:42.092141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e729f544-ca93-4868-98a0-fec7a28147a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:42.235520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6fe9a4-72a8-49cf-8ecf-77a585a97198 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:42.260491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:42.267791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 353/686] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 220 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:42.291692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:42.292705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.293210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.293680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.299122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62c25af-2de6-432c-87d5-738f38dfb4ea None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.299666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62c25af-2de6-432c-87d5-738f38dfb4ea None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.300139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62c25af-2de6-432c-87d5-738f38dfb4ea None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.301975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 354/687] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:42.309031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.309713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.310058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.338928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:42.340461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:42.387940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:42.389829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:42.426547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:42.434954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 355/688] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:42.509465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 334/689] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:42 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 481 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:17:42.514861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:42.562991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:42.586293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:42.631061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a182fbbe-c83e-4b12-b208-9454c621d8a9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:42.669175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:42.670325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.670701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.671092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.683030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:42.685466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:42.704318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:42.705477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:42.706555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:42.706555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:42.706555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:42.711083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 356/690] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:42.714381 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:42.754317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:42.761978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6124a54-5822-48ca-8100-4b0075a3d579 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:42.798437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:42.799010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.799292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.799569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.810747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:42.815617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:42.816543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:42.817399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 335/691] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:42.818107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:42.848482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e9d58d1-c227-432f-975f-9f1936a05061 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:42.890868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:42.891596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.891946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.892289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:42.893175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:42.894347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:42.894876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:42.895959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:42.909712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 357/692] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:42.910571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:42.938275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe3a36ef-b466-4ca0-97a5-091cfb9d5ed8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:42.976498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:42.977388 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:42.977666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:42.977933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.016432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:43.037795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed5431f9-34c8-449e-979f-13b1b7efbc39 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.077028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.077827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.078102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.078370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.079074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:43.079376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.080751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.095830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 358/693] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:43.095830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.116606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.118568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:43.134760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:43.135815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:43.136701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:43.136701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:43.136701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:43.138709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 336/694] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:42 2022] GET /identity/v3/domains/Default => generated 87 bytes in 321 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:43.139727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.196342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991990df-7c00-46ef-9601-62639a868f2d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.234975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.235858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.236208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.236837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.245169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.247371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.252504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff0de30-3442-4923-9c2a-eaa6f1315d5c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:43.261156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:43.262232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:43.263349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:43.264886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 359/695] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:43.265736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.275214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:43.287188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:43.299486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.300545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.300906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.301391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.315926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.317479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:43.341412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:43.342557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:43.343644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:43.343644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:43.343644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:43.343644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 337/696] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:43.347865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.401714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7a7f5d-fb64-4a85-86b4-b0db30af5495 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.439414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.440025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.440305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.440575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.441204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:43.441505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.442982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.461155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 360/697] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:43.462149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.491284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35dd3622-aa4d-49d9-9e07-6d4a14513d76 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.537694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:43.542653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.543399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.543687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.544466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.544736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:43.545093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.546555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.566260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 338/698] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:43.567189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.586064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f884b4e1-b9ce-49f5-a342-af12a1c64f29 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.605915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:43.636366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.637577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.638047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.638620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.640512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:43.641005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.642630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.660178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 361/699] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:43.661902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.688108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e96a8fad-5acd-48c2-9432-f650d524bd3b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.737415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff9746d6-c27d-42f7-9c3b-30af18235c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.738877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9746d6-c27d-42f7-9c3b-30af18235c06 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.739196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9746d6-c27d-42f7-9c3b-30af18235c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.740520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff9746d6-c27d-42f7-9c3b-30af18235c06 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.741176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9746d6-c27d-42f7-9c3b-30af18235c06 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.742754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff9746d6-c27d-42f7-9c3b-30af18235c06 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.780060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac627a1-d003-48f1-ae7c-1de38b9605e6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:43.809552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:43.821773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:43.822430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.822715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.823014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.823795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=85bbaf87748242bf830ca84267692c82) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:43.824126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:43.825609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:43.838856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 362/700] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/role_assignments?user.id=85bbaf87748242bf830ca84267692c82&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:43.838856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:43.868348 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 339/701] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:17:43 2022] POST /identity/v3/users => generated 254 bytes in 301 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:43.907945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd0e5e9-efc2-4bf3-8efb-0d4dde793d20 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.908272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd0e5e9-efc2-4bf3-8efb-0d4dde793d20 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.908524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd0e5e9-efc2-4bf3-8efb-0d4dde793d20 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.910445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 340/702] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:43.916835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:43.917267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:43.917684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:43.954037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:43.955543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:44.032263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-400360c9-bef8-4e22-ad18-f861b16b6fff None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:44.100230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:44.101392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.101996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.102407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 341/703] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:44.102791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.116398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:44.118558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:44.130976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:44.132104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:44.133676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 363/704] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:17:43 2022] GET /identity/v3/users/nova => generated 82 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:44.139850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:44.164799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:44.249309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c535762-08fa-46c5-ab49-1224a2879019 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:44.286280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:44.286991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.287268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.287537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.288261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:44.288625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:44.290209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:44.309499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 342/705] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:17:44 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:44.314612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:44.372123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:44.396995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:44.442584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6467ef31-6e60-4895-a939-507d6b343062 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:44.509920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb735f66-915e-4c3c-84a2-58fbe8dfdeb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:44.510535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb735f66-915e-4c3c-84a2-58fbe8dfdeb5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.510837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb735f66-915e-4c3c-84a2-58fbe8dfdeb5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.511109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb735f66-915e-4c3c-84a2-58fbe8dfdeb5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.556143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:44.597275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845709fd-6fd5-4553-bccb-81748b82020e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:44.645475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb735f66-915e-4c3c-84a2-58fbe8dfdeb5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:44.648134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb735f66-915e-4c3c-84a2-58fbe8dfdeb5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:44.649235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:44.650176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.650667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.651160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] PATH_INFO: `/v3/services/network` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.651964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:44.655943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:44.674568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:44.675678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:44.676501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:44.679402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 343/706] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:44 2022] GET /identity/v3/services/network => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:44.685983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:44.740081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:44.812805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 364/707] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:44 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Sep 27 16:17:44.814903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23964bca-8d1f-4633-ba3b-2d3fe3523c9e None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.815222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23964bca-8d1f-4633-ba3b-2d3fe3523c9e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.815496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23964bca-8d1f-4633-ba3b-2d3fe3523c9e None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.816968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 365/708] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:44.818690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23964bca-8d1f-4633-ba3b-2d3fe3523c9e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:44.834373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8cf8ca7-ce1f-4ca1-a593-8de5d219d663 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:44.873104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:44.873847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.874116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.874392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.875050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] RBAC: Adding query filter params (name=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:44.875357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:44.876850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d5388e-f8a1-4f61-a648-6644df8398e2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:44.882600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 344/709] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:17:44 2022] GET /identity/v3/services?name=network => generated 124 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:44.883878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:44.884168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:44.884443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:44.899322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:44.905966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:44.907211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:44.959199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23964bca-8d1f-4633-ba3b-2d3fe3523c9e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:44.987127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 345/710] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:44.988079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.011459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.012489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.013024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.013550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.035801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.039634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.059632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:45.072001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:45.086579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:45.087523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:45.087523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:45.087523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:45.087523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 366/711] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 259 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:45.087523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.158740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c12d0c54-7d39-49e0-95be-690c2c873c37 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.206361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.206993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.207275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.207546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.208277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] RBAC: Adding query filter params (type=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:45.208582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.210065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.213843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:45.223206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 346/712] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:17:44 2022] GET /identity/v3/services?type=network => generated 124 bytes in 235 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:45.234863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4539c052-caaf-4ec2-ba83-726873bed63b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.272458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.273739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.274056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.274425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.275002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:45.275307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.276680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.290258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 367/713] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:45.295723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.368119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.449536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a29723-97b8-4bf7-a130-4cc96cbc85d2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.507187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.508616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.508895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.509164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.515242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:45.517714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.519674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:45.533205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:45.534253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:45.535164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:45.538276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 347/714] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:45.545064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.603891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0ac505-334f-4794-9c9f-92a88ead82d3 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.653316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:45.667514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.668160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.668434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.668699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.669278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.670642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:45.690630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:45.691746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:45.692580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 368/715] 10.222.0.64 () {56 vars in 1129 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 325 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:45.699737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.732742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47aa2d43-c749-4922-927b-8b608a9fc3a5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.791332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.792062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.792339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.792615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.793315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:45.793623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.795083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.809926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 348/716] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:45.815486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:45.872994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbf43461-d456-4ad9-bfe6-217d7fcfbcba None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.928202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:45.963587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:45.964125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:45.964406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:45.964675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:45.965305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:45.965607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:45.967001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:45.975507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-257be409-6aaf-49bf-bc8e-253ce47a7267 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:45.977569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 369/717] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 278 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:45.983031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:46.008474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:46.034512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:46.035150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:46.035426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:46.035696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:46.036417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=ab556e3087544c42929aab1b35d145a4) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:46.036724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:46.044186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35acff6-b9d7-4d75-bcf0-1a4c54ee4f5a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:46.057835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 349/718] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/role_assignments?user.id=ab556e3087544c42929aab1b35d145a4&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 243 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:46.126999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-609801e2-899e-4ebd-977c-e5bd497ac62a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:46.169911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:46.170535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:46.170839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:46.171113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:46.171745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:46.172046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:46.173412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de384b12-02c6-466c-9815-538e191fc2a6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:46.186973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 370/719] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Sep 27 16:17:45 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 197 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:46.622296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf942ce6-fecc-46ea-b370-084f6aa741c4 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:46.623011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf942ce6-fecc-46ea-b370-084f6aa741c4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:46.623502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf942ce6-fecc-46ea-b370-084f6aa741c4 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:46.625282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 350/720] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:46 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:46.631023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:46.631587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:46.632081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:46.636283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744bf0a2-b104-46b3-9002-70e075910d94 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:46.636800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744bf0a2-b104-46b3-9002-70e075910d94 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:46.637288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-744bf0a2-b104-46b3-9002-70e075910d94 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:46.638802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 351/721] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:46.644317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:46.644596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:46.644893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:46.658456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:46.660885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:46.680477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:46.682340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:46.750695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:46.792560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 371/722] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Sep 27 16:17:46.806064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 352/723] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:47.090418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.119473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.264265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fa9f994-5fa1-40ae-8753-6a0c54447fa1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:47.307905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b27bee5-9aa3-4926-b035-7bfac58c9e0e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:47.319112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:47.320132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.320608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.321069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.332169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:47.338011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:47.338992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:47.339836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:47.342685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 372/724] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/roles/admin => generated 83 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:47.352840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.361447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:47.391057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:47.391699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.391977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.392247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.409296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:47.414342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:47.415431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:47.416747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 353/725] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/roles/service => generated 85 bytes in 297 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:47.421830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.463173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:47.544983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a5a0f23-6167-45ec-8e57-581fda11db26 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:47.560864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6adb99-0e73-412b-b172-6105c9c1176f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:47.580850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:47.581463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.581745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.582020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.582705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:47.583032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:47.584410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:47.593660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 354/726] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:47.602589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.625571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:47.626175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.626449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.626713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.627401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:47.627698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:47.629059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:47.651090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 373/727] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:47.654455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:47.658215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.746978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b40220-fc7f-47e9-add7-8cb6be523c32 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:47.808384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:47.809016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.810830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.811108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.833997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:47.835581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:47.851842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48d6d01a-e739-4abf-bb8b-554ed54696e0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:47.855177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:47.864043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:47.865016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:47.866295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:47.866295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:47.866295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:47.866295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 355/728] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:47.875435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.907255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:47.908015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.908296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.908569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.920741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:47.922348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-5b803798-40e3-497d-9c55-ec05ea1c900e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:47.939472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:47.941048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:47.942242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:47.942242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:47.942242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:47.942242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 374/729] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:47.942517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec80784f-44cb-45dd-b9c4-381d8b9ae8da None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:47.942599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec80784f-44cb-45dd-b9c4-381d8b9ae8da None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:47.944222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec80784f-44cb-45dd-b9c4-381d8b9ae8da None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:47.946150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 375/730] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:47.947405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec80784f-44cb-45dd-b9c4-381d8b9ae8da None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:47.979902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:47.995342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12404d7e-46c9-4c49-94da-99f771bfb8a0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.034674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.035778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.036237 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.036687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.037500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:48.037982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.040301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe00c00-0c48-4eb9-91f3-9ac08a6a6373 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.054953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec80784f-44cb-45dd-b9c4-381d8b9ae8da None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.070363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 356/731] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:48.070363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.070363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.070363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.093596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.094197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.094471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.094735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.095417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:48.095713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.097058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.100344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:48.103001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:48.110729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 376/732] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:48.111696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.135105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:48.208033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 357/733] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:48 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:48.208033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.248968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afd9a504-2aa4-4d87-8642-0d61283117ed None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.320779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.322160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.322441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.322757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.336052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.337387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:48.362406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:48.363443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:17:48.364386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:48.365576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 377/734] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:17:48 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 254 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:48.372577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:48.373054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.412388 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba79aae-8240-4aca-a794-c1f819972b0d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.490035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.495585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.501938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.501938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.511877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.513327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.522445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:48.540815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:48.541816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:48.542758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 358/735] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:48 2022] GET /identity/v3/users/swift => generated 83 bytes in 337 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:48.543379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.559870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a619d42a-b2f6-4576-b3d2-c976768b716c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.605274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.606512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.606867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.607097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.607828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:48.608137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.611114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.631241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 378/736] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Sep 27 16:17:48 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:48.632987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.668327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15df7408-ef5a-4870-831c-cc0d3ac8f692 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.714304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ee0c304-3d2c-46e1-b439-e1cbaad1698c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.715372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee0c304-3d2c-46e1-b439-e1cbaad1698c None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.715853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee0c304-3d2c-46e1-b439-e1cbaad1698c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.716343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ee0c304-3d2c-46e1-b439-e1cbaad1698c None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.717188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee0c304-3d2c-46e1-b439-e1cbaad1698c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.718857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ee0c304-3d2c-46e1-b439-e1cbaad1698c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.756151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e84e48-2c97-41ed-9bbe-9649129ae3b6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.792170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.792778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.793058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.793326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.794047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:48.794351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:48.795813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:48.797599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:48.809604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:48.818949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 359/738] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:17:48 2022] POST /identity/v3/services => generated 240 bytes in 273 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:48.819157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 379/738] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Sep 27 16:17:48 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:48.819157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.819331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be80581-8112-435c-bfa4-10275fb7bdae None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:48.819331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be80581-8112-435c-bfa4-10275fb7bdae None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:48.819331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be80581-8112-435c-bfa4-10275fb7bdae None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:48.824543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 360/739] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:48.832041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be80581-8112-435c-bfa4-10275fb7bdae None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:48.947593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0285a62e-b89c-46a8-96be-f61d56158650 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.999197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be80581-8112-435c-bfa4-10275fb7bdae None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:48.999438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e1ba498-7a6d-4fd3-bf11-15b2aa68fae8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:48.999967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1ba498-7a6d-4fd3-bf11-15b2aa68fae8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.000247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1ba498-7a6d-4fd3-bf11-15b2aa68fae8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.000516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e1ba498-7a6d-4fd3-bf11-15b2aa68fae8 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.040026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b5f3d90-6ddb-401d-b193-ef4ad5a84083 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.040802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5f3d90-6ddb-401d-b193-ef4ad5a84083 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.041189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5f3d90-6ddb-401d-b193-ef4ad5a84083 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.041564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b5f3d90-6ddb-401d-b193-ef4ad5a84083 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.064417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.076319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1ba498-7a6d-4fd3-bf11-15b2aa68fae8 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.078134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e1ba498-7a6d-4fd3-bf11-15b2aa68fae8 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.107530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.115492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5f3d90-6ddb-401d-b193-ef4ad5a84083 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.117540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b5f3d90-6ddb-401d-b193-ef4ad5a84083 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.185959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 380/740] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:48 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:49.185959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.185959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.185959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.204177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:49.205541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:49.225066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 361/741] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:48 2022] GET /identity/v3/auth/tokens => generated 898 bytes in 393 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Sep 27 16:17:49.227050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f2fb26-9084-4d12-a146-48f265b3f6a8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.227329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f2fb26-9084-4d12-a146-48f265b3f6a8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.227593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f2fb26-9084-4d12-a146-48f265b3f6a8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.228950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 362/742] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:49.233601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f2fb26-9084-4d12-a146-48f265b3f6a8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.282390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 381/743] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:49 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:49.283275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.304205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.357241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.365243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f2fb26-9084-4d12-a146-48f265b3f6a8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:49.397226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fdab32-43d2-4c9c-b1e5-5f41b3014311 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:49.422276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.423417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.424128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.425044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.437081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.437717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.438131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.438407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.439074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.441539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.449546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.451127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a83b7ea1-4496-4e4b-a4dc-db057e9844be None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:49.465213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:49.466172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:49.467116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:49.467116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:49.467116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:49.467116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 363/744] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 234 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:49.467116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.467116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:49.467469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:49.468331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:49.469990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:49.469990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:49.469990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:49.470508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.472993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 382/745] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:49.473984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.496013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:49.497617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:49.552047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.586418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c69f62fd-e647-40fe-aee6-cd984267d4fe None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:49.594873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 364/746] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:49 2022] POST /identity/v3/auth/tokens => generated 898 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Sep 27 16:17:49.594873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.628627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.629240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.629530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.629806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.630442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:49.630747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.633113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.647871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 383/747] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:49.649004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.680295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.707147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1d9276-ba5c-4e9d-aea6-cbecc7afc982 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:49.743992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.744930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.745411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.745881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.746734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:49.747268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.749014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.761178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 365/748] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:49.762264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.781440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e78b8d36-fb17-4468-9b3b-8605506621f6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:49.822142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.830325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.831236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.831527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.831855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.832592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.833967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.871853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e20a5a-7b75-45f1-8881-7046512c24e2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:49.904847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:49.905470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:49.905743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:49.906010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:49.908599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:49.913040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:49.915259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:49.918263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 384/749] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:17:49 2022] POST /identity/v3/services => generated 251 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:49.927086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:49.928223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:49.929874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 366/750] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:49.930619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:49.931626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.056148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f55b8a26-9c0e-40bc-8d2f-df47d7438008 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.075453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e5e0f9-ee2c-4e3d-8431-064567b9ce1c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.095907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.096766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.097152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.097516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.110325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:50.115708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:50.117637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:50.118698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:50.122148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 367/751] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:50.123027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.123997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.124802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.125091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.125370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.135892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.137835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:50.153796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:50.154804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:50.155736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 385/752] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:50.156229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.192031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:50.224778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:50.247054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-286c553c-9690-4660-8267-c094ec11d802 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.289818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a48d2181-4024-4df8-8820-681043b3a712 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.293540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.294274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.294654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.295056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.295853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:50.296261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.297829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.312415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 368/753] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:50.313509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.340100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.341129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.343316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.343624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.344899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:50.345315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.347238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.359737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 386/754] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:50.360825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.434015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feea7c9e-2eae-4380-ba4d-1baf07deb361 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.452877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:50.472492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.473322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.473762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.474186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.475148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:50.475734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.477412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.485685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1827751b-3f4a-4c45-8147-79adead253c4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.492441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 369/755] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:50.493728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.496734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:50.530948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.531849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.532235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.532920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.533994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=b821384a2058464892f0a0f37f34b914) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:50.534418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.535945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.546866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 387/756] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/role_assignments?user.id=b821384a2058464892f0a0f37f34b914&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:50.548915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.654385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dda03dc-5f22-4141-8e94-17947509d850 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.720704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.721415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.721736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.721971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.735756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.737223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:50.755789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:50.756814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:50.758604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:50.758604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:50.758604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:50.758604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 370/757] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 264 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:50.765252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:50.773540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3045264-9f16-46d3-ac7e-fa41ac982e87 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.811501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.812111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.812390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.812721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.813518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=85bbaf87748242bf830ca84267692c82) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:50.813827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.815366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.821663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 388/758] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/role_assignments?user.id=85bbaf87748242bf830ca84267692c82&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:50.833655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:50.915203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8143cf74-f13e-444f-b2b5-569fe5fe4c59 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:50.964519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:50.965154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:50.965433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:50.965704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:50.966362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:50.966666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:50.968208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:50.982042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 371/759] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:50.987943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:51.029017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:51.104244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7cd1c473-327f-499f-8a09-8d075165520d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:51.142034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:51.142688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:51.143007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:51.143279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:51.152675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:51.154175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-72d54495-d47c-4ae5-aa43-01700cdc7c38 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:51.167691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:51.169632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:51.170767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 389/760] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:17:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:51.177070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:51.257487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9791f-7a15-4102-b16a-f0b40f23f80a None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:51.257960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9791f-7a15-4102-b16a-f0b40f23f80a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:51.258229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfb9791f-7a15-4102-b16a-f0b40f23f80a None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:51.259825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 390/761] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:17:51.265491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:51.265771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:51.266047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:51.285034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:51.287064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:51.289136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:51.298573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b81b90-b34a-444c-8905-71935cca6215 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:51.348628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:51.349397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:51.349739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:51.350074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:51.351437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:51.352434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:51.354793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:51.375371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 372/762] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:17:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:51.381085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:51.401494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 391/763] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:51 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:17:51.502106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d0bcfe8-aeb7-4567-acd1-3379948d9150 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:51.549936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:51.550580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:51.550889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:51.551166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:51.567022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:51.568491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:51.580552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:51.590235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:51.594750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:51.595697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:51.595697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:51.595697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:51.595697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 373/764] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:51.597006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:51.627109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:51.707580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b01eb5c-7b60-437a-9258-379839e98459 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:51.712656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:51.747161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:51.747783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:51.748066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:51.748343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:51.748987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:51.749296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:51.750765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:51.763478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 392/765] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:51.769121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:51.923343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4903c7f-e74f-4628-b237-eb708c256414 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:51.942763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:51.974996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdbeb76-92bc-4e29-af8c-e1f6a39131c0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:51.999581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.000220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.000497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.000774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] PATH_INFO: `/v3/services/network` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.001359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.002725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:52.015754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:52.016755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:52.017679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 374/766] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:51 2022] GET /identity/v3/services/network => generated 88 bytes in 305 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:52.025345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:52.039291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.040277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.040759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.041247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.048481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.050239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:52.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:52.063556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:52.064464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:52.067033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 393/767] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:52.080173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:52.121201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:52.197421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc63f5f3-5a2d-47b7-ac17-27594b450e65 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:52.265499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.266184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.266394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.266661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.267313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] RBAC: Adding query filter params (name=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:52.267618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.268984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.284777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 375/768] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:17:52 2022] GET /identity/v3/services?name=network => generated 124 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:52.285287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d5c4e5-9afb-4c95-9f99-3222f429fc1e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:52.290761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:52.327890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.328514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.328787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.329056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.329751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:52.330053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.331511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.366945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 394/769] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:52.369595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:52.384622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:52.469036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf399e-a540-4b2d-8ce2-908f5bd7bdfd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:52.524432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:52.536021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.536456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.536736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.537006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.537638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] RBAC: Adding query filter params (type=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:52.537938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.539762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3524743-b791-43fa-94ab-885d697d10dd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.545134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 376/770] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:17:52 2022] GET /identity/v3/services?type=network => generated 350 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:52.546305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f12f50-fd14-4726-9828-1c2c7ab40a88 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.546584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f12f50-fd14-4726-9828-1c2c7ab40a88 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.546927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f12f50-fd14-4726-9828-1c2c7ab40a88 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.548123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 377/771] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:52.553132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f12f50-fd14-4726-9828-1c2c7ab40a88 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:52.572815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4b49eec-95c8-42fc-af4c-e3a3648cc503 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:52.619714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de2e9766-46dd-448e-a933-905ea94d4755 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.621679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2e9766-46dd-448e-a933-905ea94d4755 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.622116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2e9766-46dd-448e-a933-905ea94d4755 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.622820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de2e9766-46dd-448e-a933-905ea94d4755 None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/ab556e3087544c42929aab1b35d145a4/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.624883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2e9766-46dd-448e-a933-905ea94d4755 None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=ab556e3087544c42929aab1b35d145a4, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.627383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de2e9766-46dd-448e-a933-905ea94d4755 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.662302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f12f50-fd14-4726-9828-1c2c7ab40a88 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:52.705475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.706443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.706960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.707457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.707561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 395/772] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:52 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/ab556e3087544c42929aab1b35d145a4/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:52.708468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] RBAC: Adding query filter params (interface=public, service_id=63be8283d9674af99003f1c5ecc7f104, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:52.709008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.709115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.709282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.709605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.711657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.721153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 378/773] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:17:52 2022] GET /identity/v3/endpoints?service_id=63be8283d9674af99003f1c5ecc7f104&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:52.730826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:52.763131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:52.764339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:52.818894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:52.897193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14be3266-3016-41b2-bdef-a13070037cb7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:52.919481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:52.930010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 396/774] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:52 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:52.941977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-098d50ee-e740-4778-81d7-73fcd8939cd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:52.942586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d50ee-e740-4778-81d7-73fcd8939cd6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:52.942916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d50ee-e740-4778-81d7-73fcd8939cd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:52.943192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-098d50ee-e740-4778-81d7-73fcd8939cd6 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:52.943802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098d50ee-e740-4778-81d7-73fcd8939cd6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:52.945746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-098d50ee-e740-4778-81d7-73fcd8939cd6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:52.951751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 379/775] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:17:52 2022] GET /identity/v3/services => generated 772 bytes in 226 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:53.295887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:53.397260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221acf89-5eb7-4c69-9a87-b33d8ec65d63 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.398669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221acf89-5eb7-4c69-9a87-b33d8ec65d63 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.399172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-221acf89-5eb7-4c69-9a87-b33d8ec65d63 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.402762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 380/776] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:53 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:53.409675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bdb194-08f8-46f0-9f69-c7dc2daeb234 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.410214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bdb194-08f8-46f0-9f69-c7dc2daeb234 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.410721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bdb194-08f8-46f0-9f69-c7dc2daeb234 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.431752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b1bdb194-08f8-46f0-9f69-c7dc2daeb234 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:53.433969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b1bdb194-08f8-46f0-9f69-c7dc2daeb234 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:53.481697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9830d360-6cf4-4943-a3bb-42ce3105a234 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:53.522422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 381/777] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:17:53.522422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6adf02-0ca5-4b87-abf7-84604ab992e4 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.522422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6adf02-0ca5-4b87-abf7-84604ab992e4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.522422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd6adf02-0ca5-4b87-abf7-84604ab992e4 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.522422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 382/778] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:53.528529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.528808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.529080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.540307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:53.540951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.541229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.541501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.542153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:53.543566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:53.561083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:53.562209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:53.566835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:53.567833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:53.568698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:53.572744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 397/779] 10.222.0.64 () {56 vars in 1129 bytes} [Tue Sep 27 16:17:53 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 277 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:53.583772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:53.617578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:53.643577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 383/780] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:53 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:53.696254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:53.731148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb695c7-31e2-448b-b7e2-1a4bdaba903c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:53.775881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:53.776494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.776771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.777040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.777667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:53.777967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:53.779355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:53.788723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 398/781] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Sep 27 16:17:53 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:53.791935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:53.904914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:53.938682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c813bf-d58f-45f6-be34-e05d327dbf61 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:53.984546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:53.985424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:53.985701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:53.985987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:53.986849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:53.987533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:53.988903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:53.996367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 384/782] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Sep 27 16:17:53 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:53.997762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.046244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.049699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c50359-6cc1-4cbc-adf4-5020c19a06d6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.087368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.088107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.088397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.088676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.099418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:54.105621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:54.106558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:54.107453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 399/783] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:54.108144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.116564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12f61f63-0023-4b6d-bb3c-e7051153739a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.135168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.162085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.162938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.163411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.163687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.176752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:54.183474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:54.185272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 385/784] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:54.186636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.232814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9f842c-7b78-4fd0-ab96-965acf241d69 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.270197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.270865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.271147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.271408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.272101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] RBAC: Adding query filter params (interface=public, service_id=60688d5ad6c14194a85ec93e3ebd78f7, region_id=RegionOne) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:54.273231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.274545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.285393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 400/785] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/endpoints?service_id=60688d5ad6c14194a85ec93e3ebd78f7&interface=public®ion_id=RegionOne => generated 194 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:54.286233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.307205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b88870-d4cf-4e5c-b3d6-2cd3b00d1da4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.319122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.336678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.352658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.353383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.353759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.354148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.354939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:54.355713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.357293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.367199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 386/786] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:54.368358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.392925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1ea364-67c1-44de-b7e2-1e92d6463b90 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.427842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.428486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.428913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.429254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.429978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:54.430346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.431985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.440538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 401/787] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:54.441346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.478564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cb62d1f-a413-440d-89fe-f9a1ac4a91f3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.516436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.517351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.517772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.518235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.519005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.520561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.526481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 387/788] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/services => generated 772 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:54.527573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.545660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eef04fa-8b46-4851-9f5e-5244e940cce5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.566801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.568325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.586656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.586656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.587011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.587011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.600360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.601898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:54.625800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:54.626875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:54.627820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:54.627820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:54.627820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:54.630579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 402/789] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:54.642816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.743840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a43252-c3ab-4a83-bdcc-1510a4f9cb5f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.819870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.820529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.820813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.821089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.832614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.835790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:54.853184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:54.854171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:54.855485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:54.855485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:54.855485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:54.855485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 388/790] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 327 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:54.860451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.874612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-604fc747-2925-48b4-8912-df308301bea5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:54.910340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:54.910981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:54.911259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:54.911545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:54.912160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:54.912457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:54.914307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:54.926600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 403/791] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:54.932279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:54.963141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.972380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:54.994501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b1fd572-0d52-4ee1-afa7-52a84389b17f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.031929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.032938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.033457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.033930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.034793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:55.035303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.036874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.049779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 389/792] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:55.055203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f4ced2-90ab-441b-a161-ae042233705d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.057404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.094336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.095539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.099148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.099419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.108952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.110409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:55.130827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:55.131859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 404/793] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:17:54 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:55.132824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.211369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ace04c1-67db-4e53-abb1-17b8bddb44d8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.222555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:55.250064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7280547-e6d4-474e-a9af-3fbe2e818ef8 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.279470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.280119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.280456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.280728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] PATH_INFO: `/v3/users/swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.294417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.297125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.305244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:55.322234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:55.323308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:17:55.324258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:55.324777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 390/794] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/users/swift => generated 83 bytes in 267 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:55.330480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.331256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.331543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.331813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.332568 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:55.332876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.334363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.339171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.378545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 405/795] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 246 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:55.384090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.494155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-564e4fb8-1d59-4891-8f85-86193141a19d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.512080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a289e25-e583-4b64-88ec-9a0b87ab5e26 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.533064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.533710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.533945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.534214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.535025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:55.535304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.536706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970eff63-33cb-47f5-931c-7d00371edcaf None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.559806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 391/796] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:55.559806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc08af6-d6b5-4aa9-982d-6318258e9c70 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.559806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc08af6-d6b5-4aa9-982d-6318258e9c70 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.559806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdc08af6-d6b5-4aa9-982d-6318258e9c70 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.559806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 392/797] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:55 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:55.559806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a63473-f871-42ca-9e52-f7d115085cc7 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.560300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a63473-f871-42ca-9e52-f7d115085cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.560300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a63473-f871-42ca-9e52-f7d115085cc7 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.561559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 393/798] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:55.562892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a63473-f871-42ca-9e52-f7d115085cc7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.565751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.566383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.566691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.566985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.576382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:55.578843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.580275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-6e4984e9-6d1b-4398-95cf-9c6b7b481962 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:55.595888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:55.596862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:55.597784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:55.597784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:55.597784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:55.597784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 406/799] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 214 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:55.598502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d53efd-1437-429b-b6d9-746f158a8e7d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.598675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d53efd-1437-429b-b6d9-746f158a8e7d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.598972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79d53efd-1437-429b-b6d9-746f158a8e7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.617928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-79d53efd-1437-429b-b6d9-746f158a8e7d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:55.619145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-79d53efd-1437-429b-b6d9-746f158a8e7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:55.636364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:55.681407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a63473-f871-42ca-9e52-f7d115085cc7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.692022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 407/800] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:55.693291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.693568 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.693833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.714664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:55.715888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:55.726516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:55.727617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:55.728061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:55.728495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:55.741452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:55.743429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:55.759272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:55.760324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:55.761221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:55.761221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:55.761221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:55.763006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 394/801] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:55.764067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.798800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 408/802] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:55 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:55.798800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:55.853205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:55.939698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eca114ef-8395-4e09-a9b0-250d4e3a953f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.945823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed3abe39-cbd6-4e8d-9ac9-ec891d97c52d None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:55.963256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.001131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.001805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.002083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.002347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.003017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:56.003320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.004672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.016085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.016666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.016920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.017190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.017389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 409/803] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:56.017840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:56.018142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.026831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.028059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.032097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 395/804] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 268 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:56.033013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.137962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-328bb600-4cca-4ed6-94f9-33e1c12ed070 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.151709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.161620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6125cc5-66dc-461f-9a43-38e15a69a799 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.172452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.173171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.173448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.173730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.180914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.182340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:56.195727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:56.196621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:56.197505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:56.199672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.200305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.200580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.200845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.201653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 410/805] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:56.202505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.213093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:56.219725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:56.220655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:56.221477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:56.222111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 396/806] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:56.222823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.263378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.307577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef482244-6e25-40a5-b5a6-7206882a92f4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.345615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2e5b8f3-0e52-48e6-aa5a-6ca76a19f0f6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.346500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.347157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.347432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.347698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.355312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.357041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:56.370384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:56.371561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 411/807] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:56.372457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.387050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.387955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.388250 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.388504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] PATH_INFO: `/v3/services/network` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.389084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.390452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:56.400027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:56.401295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Sep 27 16:17:56.402328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:56.406001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 397/808] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/services/network => generated 88 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:56.406840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.410071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.482059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-047afda8-d8bb-494d-bbc7-0738732f7d90 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.508125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.514364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-509edf44-d39d-46a6-93b8-143978cf39f5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.516776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.517364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.517642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.517907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.518582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:56.518911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.520332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.536631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 412/809] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:56.536631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.558921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.559541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.559821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.560102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.560749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:56.561049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.570819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 398/810] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:56.571362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.636825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.646304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373d595f-b5fd-4727-a59c-12db0c4f170e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.670198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-552cc6c3-1ce5-451d-a1a8-d1c6aa87fe73 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.684133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.684860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.685137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.685403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.686087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:56.686386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.688069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.705700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 413/811] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:56.705700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.711115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.711720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.711996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.712262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.712884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] RBAC: Adding query filter params (name=network) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:56.713182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.714541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.722868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 399/812] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/services?name=network => generated 124 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:56.722868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.743886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.816317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33990ca3-90c9-4775-a8cc-bf3861ee715a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.822902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e926b012-5473-4d34-bc02-cf078124ccb6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:56.855029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.855684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.855960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.856228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/85bbaf87748242bf830ca84267692c82/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.858090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=85bbaf87748242bf830ca84267692c82, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.859206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:56.859530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.860224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:56.860678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:56.861119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:56.873092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:56.874528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:56.889823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:56.890985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:56.891887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:56.891887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:56.891887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:56.891887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 400/813] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:56.891887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.943124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 414/814] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:56 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/85bbaf87748242bf830ca84267692c82/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 231 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:56.943124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:56.961321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:56.993848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:57.027636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5723e751-bf50-4a9a-bfff-d9188c77907e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:57.067598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:57.068216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.068496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.068776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/b821384a2058464892f0a0f37f34b914/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.080136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013f0d2d-17d5-472d-adbb-a6fadbd58d4e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:57.087721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=b821384a2058464892f0a0f37f34b914, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:57.089324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:57.123457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:57.124492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.124978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.125454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.126300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] RBAC: Adding query filter params (type=network) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:57.126895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:57.128535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:57.134920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 415/815] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:17:56 2022] GET /identity/v3/services?type=network => generated 350 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:57.135919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:57.161392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 401/816] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:17:56 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/b821384a2058464892f0a0f37f34b914/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:17:57.162429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:57.234729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:57.264742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1c08ef3-a1c2-4fa6-adda-c0cb548c5a82 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:57.315903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:57.316777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.317438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.317780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.318719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:57.319064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:57.321189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b941e7c3-edd3-484c-a83e-36739c15459c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:57.321724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca44e7d7-f464-49e2-b8f2-ffccb1146c11 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:57.344548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 416/817] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:57.350688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d563cd-c412-42ed-aa97-5179d5179be1 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.351618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d563cd-c412-42ed-aa97-5179d5179be1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.351910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d563cd-c412-42ed-aa97-5179d5179be1 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.353296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:57.353690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 417/818] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:57 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:57.354711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d563cd-c412-42ed-aa97-5179d5179be1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:57.367742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7adaefc3-1130-4acb-973c-9d5ab251f1c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:57.368617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adaefc3-1130-4acb-973c-9d5ab251f1c6 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.369139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adaefc3-1130-4acb-973c-9d5ab251f1c6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.369572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7adaefc3-1130-4acb-973c-9d5ab251f1c6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.370403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adaefc3-1130-4acb-973c-9d5ab251f1c6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:57.372045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7adaefc3-1130-4acb-973c-9d5ab251f1c6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:57.471791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d563cd-c412-42ed-aa97-5179d5179be1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:57.483681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 402/819] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:17:57 2022] POST /identity/v3/endpoints => generated 337 bytes in 321 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:17:57.487559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.487559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.487559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.519584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:57.520793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:57.568276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:57.582385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:57.583390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.584052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.585272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] PATH_INFO: `/v3/users/nova` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.602217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:57.605009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:17:57.624944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:57.626098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:17:57.627276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:57.634252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 403/820] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:57 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:57.636041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 418/821] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:17:57 2022] GET /identity/v3/users/nova => generated 82 bytes in 282 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:57.642850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:57.795162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d8aec3a-ffc0-4561-96dd-a9df8ddfb7e8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:57.805104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:57.833326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:57.833860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:57.834135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:57.834402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:57.835995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:57.836532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:57.838037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:57.857339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 404/822] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:17:57 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:57.863400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:57.914042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:57.982615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62d7134c-fd2e-4bfc-8f53-332afccd6acc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.027527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.028524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.029078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.029871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.044984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c98490-58a9-4ef1-a40d-8b017ebaf98c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.084629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.085711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.085996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.086272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.086949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.088855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.099205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:58.102268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:58.103311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Sep 27 16:17:58.104174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:58.104946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 405/823] 10.222.0.64 () {56 vars in 1129 bytes} [Tue Sep 27 16:17:57 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:17:58.110895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.112123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.113674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.150102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:58.234331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 419/824] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:17:57 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:17:58.239710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.246592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4c8141d-3ce7-44b3-8a94-2a884ece67eb None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.298808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.298808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.298808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.298808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.299215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:58.299215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.299215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.306874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 406/825] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Sep 27 16:17:58 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:17:58.310614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.357195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:58.365906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64a51dd7-d4e5-45cf-a7b7-43e893e482f9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.401590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.402208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.402484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.402750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.414418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.415596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:17:58.430426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:58.431444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:17:58.432673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:17:58.432673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:17:58.432673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:58.432673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 420/826] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:17:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:58.438556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:58.439095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.446816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b6a470-9f3e-4f94-9d63-592d8be5192f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.497363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.498751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.499422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.500536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.501340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:58.501792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.503911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.512106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 407/827] 10.222.0.64 () {56 vars in 1124 bytes} [Tue Sep 27 16:17:58 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:58.517623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.565375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18840703-6c4e-42bb-beac-b17bebaa1c14 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.601918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.602924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.603500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.603983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.604831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:58.605497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.607071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.624954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 421/828] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:17:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:58.626132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.636740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:58.644352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef140545-49fa-4cf0-9a77-6f9362d04374 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.690333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.691158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.691432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.691774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.692791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.695010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.734436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92405e4b-b163-419d-94fd-20c63e540cdf None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.780219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 408/829] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:17:58 2022] POST /identity/v3/endpoints => generated 346 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:17:58.787173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.788447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.788836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.789201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.798039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:58.799808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:17:58.813615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:58.814819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:17:58.815836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:17:58.817714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 422/830] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:17:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:58.822755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:58.841210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:58.954056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd0b5dc8-d90a-4a8c-a18e-02d9c46268f0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:58.996909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-314e761c-6cff-4093-b462-09c100279301 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:58.997890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314e761c-6cff-4093-b462-09c100279301 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:58.998391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314e761c-6cff-4093-b462-09c100279301 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:58.998905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-314e761c-6cff-4093-b462-09c100279301 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:58.999833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314e761c-6cff-4093-b462-09c100279301 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:59.000341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314e761c-6cff-4093-b462-09c100279301 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:59.002059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-314e761c-6cff-4093-b462-09c100279301 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:59.016540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 409/831] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:17:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:59.023665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:59.087023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:59.128863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ea9b609-9dc7-4250-a030-4bdec62950a2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:59.163728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:59.164358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:59.164621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:59.164798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:59.165514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=ab556e3087544c42929aab1b35d145a4) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:17:59.165818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:17:59.167382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd87673e-e4e2-4079-96f0-4cda93ddb594 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:17:59.174549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 423/832] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:17:59 2022] GET /identity/v3/role_assignments?user.id=ab556e3087544c42929aab1b35d145a4&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:17:59.304486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c490bec0-41e9-479a-a4a6-ed2aa19f6c3d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:59.304988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c490bec0-41e9-479a-a4a6-ed2aa19f6c3d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:59.304988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c490bec0-41e9-479a-a4a6-ed2aa19f6c3d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:59.306591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 410/833] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:59.312444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aafe13f-754a-4ffc-8596-f4a07e568454 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:59.312740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aafe13f-754a-4ffc-8596-f4a07e568454 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:59.312994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aafe13f-754a-4ffc-8596-f4a07e568454 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:59.337287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6aafe13f-754a-4ffc-8596-f4a07e568454 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:59.344014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6aafe13f-754a-4ffc-8596-f4a07e568454 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:59.416486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 424/834] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:59.576242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6267c5d9-90a5-407a-a8cb-b14b77f7333e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:59.576811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6267c5d9-90a5-407a-a8cb-b14b77f7333e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:59.576997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6267c5d9-90a5-407a-a8cb-b14b77f7333e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:59.578546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 411/835] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:17:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:17:59.584140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:59.584423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:59.584734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:59.605971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:17:59.607195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:17:59.658345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:59.677595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 425/836] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:17:59 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:17:59.713982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6267c5d9-90a5-407a-a8cb-b14b77f7333e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:59.800277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:17:59.829821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6267c5d9-90a5-407a-a8cb-b14b77f7333e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:17:59.865631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:17:59.866295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:17:59.866657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:17:59.866959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:17:59.877290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:17:59.882456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:17:59.883420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:17:59.884264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 412/837] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:17:59 2022] GET /identity/v3/roles/admin => generated 83 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:17:59.891339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:17:59.950080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.009852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d2f5f6e-4645-44aa-8449-95ffbcbee8f8 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.057355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.057996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.058267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.058586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.059557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:00.059867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.061915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.063889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eb0cba1-51e1-4784-a8b4-9519f3723708 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.078803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 426/838] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:17:59 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:00.082274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.113522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.114966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.115681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.116238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.119207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:00.131938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:00.137767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:00.140227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:00.141162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:18:00.142001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:00.144675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 413/839] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:17:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:00.151474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.217713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887693f7-1fe4-4e65-aafd-1fcda0d89661 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.260648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.261558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.262108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.262851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.268778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caeabafc-4f8b-43b8-858f-5b1179804a38 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.276831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.278941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:00.297922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:00.299054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:00.300185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:00.300185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:00.300185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:00.300356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 427/840] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 219 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:00.307784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.316645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.317315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.317607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.317925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.318701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:00.319716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.321909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.331751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 414/841] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:00.341013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.413797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:00.443665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:00.450245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aeb52be-baa8-4991-b71b-5c21b4cd1e28 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.460460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4409378-6192-4c56-baa6-4ccd3590fd56 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.489331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.490033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.490608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.490717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.492438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:00.492745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.494117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.496728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.497576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.498047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.498507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.513084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 428/842] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:00.515050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.516247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.517901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:00.534428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:00.535447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:00.536338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:00.536338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:00.536338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:00.538570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 415/843] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:00.546854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.638387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333d7c15-3938-454b-95b0-9c543b5dcd0c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.671923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b8097d-2f30-49e9-862a-a8ed52a5b929 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.679741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.680389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.680689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.680965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] PATH_INFO: `/v3/users/swift` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.685936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:00.691162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.692661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:00.707449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:00.708472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:00.710306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 429/844] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/users/swift => generated 83 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:00.714312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.722068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.723071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.724133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.724617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.725547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:00.726136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.728547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.736128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:00.751017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 416/845] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:00.752562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.837466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f6f4061-4f32-4e20-a13e-122bb9e142c7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.882333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.883411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.883911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.884415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.885352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:00.886026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.887980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.888945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2527dc34-b9d0-48e3-a4cd-3057f38575ea None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:00.909498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 430/846] 10.222.0.64 () {56 vars in 1135 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:00.917827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:00.942407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:00.943412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:00.943894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:00.944354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:00.958668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:00.961077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:00.982047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:00.983882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:00.985724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:00.989819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 417/847] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 237 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:01.001806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.026312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.041883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.079118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afec7cc4-417e-43cf-86e7-69b7b155d843 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.121904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704c95e1-9ec0-43b9-bc07-3c0f75afa2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.122522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704c95e1-9ec0-43b9-bc07-3c0f75afa2d4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.122819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704c95e1-9ec0-43b9-bc07-3c0f75afa2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.123156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704c95e1-9ec0-43b9-bc07-3c0f75afa2d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.129189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9aed80-1c16-4539-8bfb-340191b87846 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.170732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.171298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.171576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.171845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.172570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:01.172875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.174303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99752f47-3bba-4098-ab00-cc151df3f6df None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:01.198102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 418/848] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:01.199406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279518f4-8725-4b2f-8dc7-43bdbe40714c None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.199690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279518f4-8725-4b2f-8dc7-43bdbe40714c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.199960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-279518f4-8725-4b2f-8dc7-43bdbe40714c None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.201325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 419/849] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:01.205128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279518f4-8725-4b2f-8dc7-43bdbe40714c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.217186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704c95e1-9ec0-43b9-bc07-3c0f75afa2d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.219815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704c95e1-9ec0-43b9-bc07-3c0f75afa2d4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:01.308191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.313943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.329794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-279518f4-8725-4b2f-8dc7-43bdbe40714c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.348183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 431/850] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:00 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 433 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:01.349703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.350059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.351021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.374586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b157e4-336d-4786-b46d-2433a7d7687d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.375477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:01.375761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b157e4-336d-4786-b46d-2433a7d7687d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.376257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b157e4-336d-4786-b46d-2433a7d7687d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.376704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:01.376867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b157e4-336d-4786-b46d-2433a7d7687d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.455406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b157e4-336d-4786-b46d-2433a7d7687d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.457619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b157e4-336d-4786-b46d-2433a7d7687d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:01.473449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 432/851] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:01 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:01.473449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.576147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.593891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 420/852] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:01 2022] GET /identity/v3/auth/tokens => generated 1161 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 27 16:18:01.595060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b41fa7e-ecfa-48d7-8637-21e7f3bcb497 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.595344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b41fa7e-ecfa-48d7-8637-21e7f3bcb497 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.595612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b41fa7e-ecfa-48d7-8637-21e7f3bcb497 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.597098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 421/853] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:01.603261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b41fa7e-ecfa-48d7-8637-21e7f3bcb497 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.619825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39b63c16-3120-45bf-9b0d-92cc3cadba55 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.629941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.661422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.662174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.662515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.663142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.674981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.676490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-3796c27a-488e-405f-b4f6-a18cdee515e2 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:01.692841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:01.694378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:01.695610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:01.695610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:01.695610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:01.695610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 433/854] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 226 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:01.697073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.697467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.697732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.719468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:01.720583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:01.726617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b41fa7e-ecfa-48d7-8637-21e7f3bcb497 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.763258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.763889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.764166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.764434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.777054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.778501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:01.793888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:01.795215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:01.796207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:01.796207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:01.796207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:01.796207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 422/855] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:01.796207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.803864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 434/856] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:01 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:01.805026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.855446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.898075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:01.909796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69783721-234d-49d5-93a3-e8a146fc9dc0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.938380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935f8f40-c88b-49ee-b9d4-30cb2a4d6f66 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:01.946575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.947235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.947519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.947789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.948369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.949744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:01.960964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:01.961999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:01.962892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 435/857] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:01 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:01.963707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:01.981880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:01.982426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:01.982706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:01.983012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:01.983651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:01.983958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:01.985481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.002373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 423/858] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.003344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.072540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7da9572-b0a9-4bc7-822e-d4fcc3df0ac9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.104848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.106207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.106874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.107156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.107455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.108098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.108403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.108966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-962bfe2c-cf70-4017-a907-0844e9e22d54 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.109766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.121993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 436/859] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.122826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.138270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.146765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.149339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.149339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.149339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.149339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.149339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.150456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.155293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 424/860] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.155917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.224246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba8040d-1b8b-476d-8895-6694e0e5d978 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.254011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0bb138-770a-4bad-9190-e00ac584ca5b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.258719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.259371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.259651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.259921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.266985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.268424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:02.279641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:02.280612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 437/861] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:02.281577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.293878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.294512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.294810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.295090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.295672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.297026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:02.306185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:02.307239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:02.308130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 425/862] 10.222.0.64 () {56 vars in 1132 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:02.308970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.337603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.360409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.389520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970e73f8-5958-47da-8520-29cb8d212b8f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.411529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a26ed147-a5a3-4067-ade8-92b76a849418 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.423206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.423855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.424131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.424398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.431499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.433035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:02.445130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:02.446206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 438/863] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:02.447544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.448088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.448088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.448088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.448285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.448842 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.449144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.451076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.455525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 426/864] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.456310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.560110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.562551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e447788-c26c-449e-bffe-c71f280a3f90 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.569876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957491fa-74e0-43e6-8031-2558b4582d0a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.596782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.602540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.603197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.603475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.603743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.604648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.604953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.606422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.611490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.612406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.612859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.613303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.614723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.615231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.618303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.620103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 439/865] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.620951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.629662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 427/866] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.631213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.726291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62682ff6-5689-446e-9627-ad9e21032972 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.743067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c14bc55b-c72f-407c-aa85-83cc939e0a1e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.759613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.760405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.760747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.761080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.761851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.762217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.763743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.775999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 440/867] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.776968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.786968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.788563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.789053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.789517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.790541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=85bbaf87748242bf830ca84267692c82) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.790949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.792510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.803140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 428/868] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/role_assignments?user.id=85bbaf87748242bf830ca84267692c82&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.804646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:02.841244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.845104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:02.910407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37b81800-9fa8-40e1-82cb-3f9af88732af None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.951648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5510a4b0-cd5c-440c-ac8b-27368567916a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:02.960142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.961399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:02.961683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:02.961957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:02.962603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:02.962945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:02.964891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6790a6bb-200c-420e-9180-166ca459c5d4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:02.971361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 441/869] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:02.998986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:02.999883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:03.000328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:03.000739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:03.001631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=b821384a2058464892f0a0f37f34b914) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:03.002091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:03.003734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27dcae14-47e3-44ae-a371-8d2dd052bea4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:03.011329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 429/870] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:02 2022] GET /identity/v3/role_assignments?user.id=b821384a2058464892f0a0f37f34b914&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:04.737686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbb0524-1070-4f6b-ab98-d0c52ac38832 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:04.738208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbb0524-1070-4f6b-ab98-d0c52ac38832 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:04.738208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbb0524-1070-4f6b-ab98-d0c52ac38832 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:04.739874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 442/871] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:04.745826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ce5336-e515-4fb1-8a69-08181d81cd98 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:04.746311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ce5336-e515-4fb1-8a69-08181d81cd98 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:04.746405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ce5336-e515-4fb1-8a69-08181d81cd98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:04.766691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-40ce5336-e515-4fb1-8a69-08181d81cd98 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:04.767945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-40ce5336-e515-4fb1-8a69-08181d81cd98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:04.843161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 430/872] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:04 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:05.049510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f97018a-1ecc-411e-8893-f78eb562655f None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.049685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f97018a-1ecc-411e-8893-f78eb562655f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.049991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f97018a-1ecc-411e-8893-f78eb562655f None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.051627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 443/873] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:05.057441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8e1bcd-91e4-41bc-8f60-a758e0caafc0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.057733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8e1bcd-91e4-41bc-8f60-a758e0caafc0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.058097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb8e1bcd-91e4-41bc-8f60-a758e0caafc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.078224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fb8e1bcd-91e4-41bc-8f60-a758e0caafc0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:05.079516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fb8e1bcd-91e4-41bc-8f60-a758e0caafc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:05.089028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:05.131279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f97018a-1ecc-411e-8893-f78eb562655f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:05.169805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 431/874] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:05 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:05.234086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a08cdb-1c3f-42dd-8150-c704a147178e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.234734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a08cdb-1c3f-42dd-8150-c704a147178e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.235258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a08cdb-1c3f-42dd-8150-c704a147178e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.237055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 432/875] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:05.242265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975634e8-d036-43e4-804b-8ea515959229 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.242829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975634e8-d036-43e4-804b-8ea515959229 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.243315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975634e8-d036-43e4-804b-8ea515959229 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.252985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f97018a-1ecc-411e-8893-f78eb562655f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:05.262542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-975634e8-d036-43e4-804b-8ea515959229 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:05.264089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-975634e8-d036-43e4-804b-8ea515959229 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:05.271019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:05.292483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:05.293093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.293373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.293644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.294407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:05.295954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:05.337245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 433/876] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:05 2022] POST /identity/v3/auth/tokens => generated 1161 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:05.338903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448cf04d-b2ad-4ce7-b1e7-310faab36afb None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.339386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448cf04d-b2ad-4ce7-b1e7-310faab36afb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.339898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448cf04d-b2ad-4ce7-b1e7-310faab36afb None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.341307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 434/877] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:05.348326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.348606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.348878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.367881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:05.369109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:05.394182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 444/878] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:05 2022] POST /identity/v3/services => generated 251 bytes in 263 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:05.446351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:05.450153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:05.458374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 435/879] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:05 2022] POST /identity/v3/auth/tokens => generated 1262 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:05.563890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e89a147-a3e7-467d-b73f-a7be3d9d8612 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:05.600954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:05.601815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.602241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.602658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.603400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:05.604918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:05.611416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:05.614098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:05.615103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:05.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 445/880] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:05 2022] GET /identity/v3/services/object-store => generated 93 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:05.621141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:05.666553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:05.738400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31be2724-a778-4dce-9ebb-2c16f1bac790 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:05.747724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b7afd5-4cf2-4be5-9d51-2156ccac3341 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:05.777686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:05.778299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.778571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.778942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.779499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:05.779798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:05.781143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:05.784774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:05.785382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.790446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.790598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 446/881] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:05 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:05.790678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.790748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:05.790841 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:05.790841 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:05.885911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecebd17f-7432-482a-9e1f-2d16725ba1ef None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:05.888082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:05.919270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:05.919902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:05.920232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:05.920508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:05.927006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 436/882] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:05 2022] POST /identity/v3/services => generated 269 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:05.927006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:05.945372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:05.951003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:05.953102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:05.954544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 447/883] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Sep 27 16:18:05 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:05.960278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:06.013416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:06.076544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a348e52b-b851-4051-ae86-7f2818563383 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:06.080561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab7c57c-552f-440c-851b-2f3718ce7d08 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:06.118151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:06.118764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:06.119077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:06.119348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:06.120621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:06.120621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:06.121731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:06.122631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:06.123310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:06.123593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:06.123862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:06.124502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:06.124812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:06.126168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:06.134156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 437/885] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:05 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:06.134866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 448/885] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:18:05 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:06.137920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:06.255717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0d63a1-f781-414c-a72e-783f402c928d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:06.276293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:06.295672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:06.296325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:06.296601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:06.296872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:06.309085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:06.310806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:06.327574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:06.328617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:06.329847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:06.329847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:06.329847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:06.329847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 438/886] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:06.336915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:06.370175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:06.450632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-755380a9-ad1a-4813-91ef-fd7e81af200a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:06.486556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:06.487187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:06.487462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:06.487728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:06.488365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:06.488662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:06.490077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:06.503666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 449/887] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:06.512097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:06.627193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bf67ce7-6e13-4d35-9823-6632911da227 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:06.679448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:06.680087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:06.680364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:06.680634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:06.688654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:06.690117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:06.704485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:06.705644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:06.706680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:06.711071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 439/888] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:18:06 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:06.711957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:06.865159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:06.882810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c781baa0-0118-4461-8bf9-f31b90d0a587 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:06.918888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:06.919879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:06.920370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:06.920849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:06.921781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:06.922319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:06.924016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:06.945845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 450/889] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Sep 27 16:18:06 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 234 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:06.952143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:06.989904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:07.103047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f00f82e8-fd7d-4e23-814d-0fd054b5bc3d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:07.159043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:07.159657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:07.159931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:07.160197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:07.298721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:07.300279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:07.339213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:07.411375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 440/890] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:06 2022] GET /identity/v3/auth/tokens => generated 1372 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:07.417680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:07.521231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-385f5de1-2290-4aa2-bc04-a42349701a52 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:07.558834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:07.559837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:07.560331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:07.560799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:07.571431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:07.573137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:07.583182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:07.589071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:07.590208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:07.591304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:07.591304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:07.591304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:07.593201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 451/891] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 176 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:07.598797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:07.704717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba39d6b-d1e8-4970-8af4-d7f26ba81d4f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:07.739473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:07.745228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:07.745839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:07.746113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:07.746380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:07.747110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:07.747413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:07.748764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aba43028-780f-49ff-8386-614247bb2494 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:07.761432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 441/892] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:07.766701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:07.811603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed216a10-089d-4d20-9e58-8597be4373f0 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:07.811788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed216a10-089d-4d20-9e58-8597be4373f0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:07.812067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed216a10-089d-4d20-9e58-8597be4373f0 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:07.813645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 442/893] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:07.819063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:07.819358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:07.819641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:07.845299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:07.846927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:07.883897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438cfbe-bc9c-4ed1-b087-dcf4e0635d65 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:07.931169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:07.931797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:07.933605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 443/894] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:07 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:07.933710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:07.933710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:07.945332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:07.946806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:07.960542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:07.962632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:07.964777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 452/895] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:07.970265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.076870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8360caa-a087-40e2-9279-7ab7f09717a6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.098106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:08.113462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.114344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.114817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.115252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.116845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:08.117310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.119047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:08.132323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 444/896] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:08.139413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.199047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.200947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:08.256883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69f303-8395-4c58-8a8c-ca8a5b1d3b60 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.307931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.308905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.309329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.309752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.310650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] RBAC: Adding query filter params (role.id=b0033600d8264a8d932bf36ce195ab0c, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=b821384a2058464892f0a0f37f34b914) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:08.311141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.311435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eb00d23-a54b-4370-ba44-0884b8b9d03a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.313378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:08.329659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 453/897] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/role_assignments?user.id=b821384a2058464892f0a0f37f34b914&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=b0033600d8264a8d932bf36ce195ab0c => generated 259 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:08.363410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.364214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.364611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.364955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.365844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.367452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4785bf77-cac3-4d98-9e7c-89f70f5458f7 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:08.377612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:08.378751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:08.379676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:08.383083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 445/898] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:08.386987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.416285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee41547-e88f-4468-affe-d283f2dca11b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.416565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee41547-e88f-4468-affe-d283f2dca11b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.417189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee41547-e88f-4468-affe-d283f2dca11b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.418907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 446/899] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:08.424822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8d00fb-be62-4a42-9120-8cb3c338047b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.425114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8d00fb-be62-4a42-9120-8cb3c338047b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.425509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b8d00fb-be62-4a42-9120-8cb3c338047b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.447000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3b8d00fb-be62-4a42-9120-8cb3c338047b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:08.448336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b8d00fb-be62-4a42-9120-8cb3c338047b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:08.501001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:08.509405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e7ff02-30f4-4bf1-8167-0405d66d431e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.532867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 447/900] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:08 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:08.533635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085d647-83a1-4e07-b9be-20e6ac847d8d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.533930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085d647-83a1-4e07-b9be-20e6ac847d8d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.534202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0085d647-83a1-4e07-b9be-20e6ac847d8d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.534497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:08.535735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 448/901] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:08.542019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.542298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.542599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.555656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.556341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.556625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.556897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.557528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:08.557835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.559258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:08.563493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:08.568972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 454/902] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:08.569891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:08.572083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.665365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 449/903] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:08 2022] POST /identity/v3/auth/tokens => generated 1372 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:08.699040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41dc4c5e-d5f6-48b7-ae0f-5e3ad09a30e0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.736853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.737511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.737793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.738116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.738820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:08.739123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.740695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:08.746514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 455/904] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:08.752484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.818203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.864144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef8a2af9-43d1-43a7-aeb4-4f32812cac2f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.866649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:08.879360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:08.901440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.902060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.902333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.902605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.903380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] RBAC: Adding query filter params (interface=public, service_id=c0a3b2773da148009e9513f96dc35835, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:08.903711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.905643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:08.917065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 450/905] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/endpoints?service_id=c0a3b2773da148009e9513f96dc35835&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:08.918169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:08.949556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd3f32e4-d0db-4f66-84f0-9cf60316ced6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:08.989011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:08.989682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:08.989956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:08.990241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:08.990869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:08.992270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:09.001079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:09.002058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:09.003789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 456/906] 10.222.0.64 () {56 vars in 1132 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:09.004591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:09.030570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e63c0d-6379-42a3-a057-314bec04bb2d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:09.068406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:09.069810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:09.070263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:09.070713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:09.071350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:09.072729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.087800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 451/907] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:18:08 2022] GET /identity/v3/services => generated 1268 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:18:09.089406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:09.127386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2aa9007-9767-4959-b778-95698f7ba008 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:09.135871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:09.145817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:09.173011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:09.173967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:09.174409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:09.174945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:09.175939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:09.177909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.223576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703e27b7-fdf6-413f-a278-3393b8a5af67 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:09.263121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 457/908] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:09 2022] POST /identity/v3/services => generated 241 bytes in 260 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:09.280941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:09.280941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:09.281228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:09.281483 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:09.282078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:09.282381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:09.283826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.290869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 452/909] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:09 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:09.296852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:09.422107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3a68370-f2fa-499a-a877-48e1476109ba None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:09.461545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:09.462084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:09.462358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:09.462625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:09.463299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:09.463604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:09.464970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.470205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 458/910] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:09 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:09.476140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:09.583979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:09.593159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8902f74e-53f0-42f2-8760-26199ffebbfa None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:09.628824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:09.629741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:09.630197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:09.630671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:09.631667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] RBAC: Adding query filter params (interface=public, service_id=b4bb01b2c090464d962210138b9f47f6, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:09.632163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:09.633723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f464498b-485b-4ab0-990d-598e423d818f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.639870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 453/911] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:18:09 2022] GET /identity/v3/endpoints?service_id=b4bb01b2c090464d962210138b9f47f6&interface=public®ion_id=RegionOne => generated 194 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:09.645806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:09.683421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:09.752393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe57bfe1-77ee-4403-bd16-2c257c607c47 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:09.789052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faa4d91e-79ff-41ea-8f9c-2b9c78b56748 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:09.789998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4d91e-79ff-41ea-8f9c-2b9c78b56748 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:09.790508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4d91e-79ff-41ea-8f9c-2b9c78b56748 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:09.790998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa4d91e-79ff-41ea-8f9c-2b9c78b56748 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:09.791852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4d91e-79ff-41ea-8f9c-2b9c78b56748 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:09.793397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faa4d91e-79ff-41ea-8f9c-2b9c78b56748 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:09.800016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 459/912] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:18:09 2022] GET /identity/v3/services => generated 1497 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:18:10.555024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07efeac-9d9c-4bd0-adc4-71f7571df403 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:10.555557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07efeac-9d9c-4bd0-adc4-71f7571df403 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:10.555557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07efeac-9d9c-4bd0-adc4-71f7571df403 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:10.557108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 454/913] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:10.562428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:10.562868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:10.563038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:10.581769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:10.582983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:10.652439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 460/914] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:10 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:10.924267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07efeac-9d9c-4bd0-adc4-71f7571df403 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.031821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07efeac-9d9c-4bd0-adc4-71f7571df403 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.069859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:11.070967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.071455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.071954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.082997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:11.089340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:11.090325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:11.091209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:11.093895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 455/915] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Sep 27 16:18:10 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:11.099611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.135917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:11.214835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93e779a3-c004-4ce5-8cd4-d702d10f6b3c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.251603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:11.252577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.253128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.253601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.254460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:11.254976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:11.256581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa50eb40-902c-4f8c-94c2-833e21e315fa None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:11.265390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 461/916] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:11.273004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.294698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:11.391122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b849341-1f58-49de-84c0-d9c8e1c6add1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.399153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489e7ba8-28ee-45d7-b202-ee796f145f1d None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.399437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489e7ba8-28ee-45d7-b202-ee796f145f1d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.399707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489e7ba8-28ee-45d7-b202-ee796f145f1d None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.401293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 462/917] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:11.406114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eabfa0-be26-4200-b496-d27feb1e8e40 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.406385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eabfa0-be26-4200-b496-d27feb1e8e40 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.407030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31eabfa0-be26-4200-b496-d27feb1e8e40 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.425323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-31eabfa0-be26-4200-b496-d27feb1e8e40 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:11.426529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-31eabfa0-be26-4200-b496-d27feb1e8e40 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:11.431233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:11.432100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.432577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.433034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.447595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:11.450338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:11.476189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:11.477176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:11.478084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:11.478084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:11.478084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:11.478084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 456/918] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:11.479095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.504748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 463/919] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:11.555169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:11.588603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaa1f18b-6387-4a8b-b104-2774dae6917b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.611724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad8973-dec5-4aed-921f-d5f3dcfa16ef None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.612246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad8973-dec5-4aed-921f-d5f3dcfa16ef None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.612246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ad8973-dec5-4aed-921f-d5f3dcfa16ef None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.614874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 464/920] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:11.619715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.619996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.620268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.624862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:11.625694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.626137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.626769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.628417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:11.628867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:11.630380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:11.638351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:11.639575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:11.648513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 457/921] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:11.650704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.661299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:11.733334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 465/922] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:11 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:11.775565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6d6ec0-211f-421b-97ee-98e1ebbb6e2b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.786211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.815661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:11.816303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.816580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.816851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.827327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:11.828772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:11.843404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:11.844361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:11.845366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 458/923] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 194 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:11.847075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:11.854462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:11.914414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-659f1643-8f7e-40a3-be15-975067f29ff8 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.955317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:11.956283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:11.956737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:11.957187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:11.958145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:11.960521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:11.971286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:11.972277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Sep 27 16:18:11.973107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:11.975209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4c1e588-b38d-48d0-8dd3-7130f01ae393 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:11.976655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 466/924] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:11.982900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.005569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.015978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.016653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.016953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.017207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.017927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.018230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.019682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.045128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 459/925] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:12.046117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.096121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.099257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62808deb-573c-4aaa-a283-f921c9467ea1 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.135412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.136066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.136291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.136564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.137189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.137629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.138881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.143875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 467/926] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:11 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:12.149225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.153609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b51f682-33a5-4d95-bb76-dc65b9e34d1b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.190294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.190954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.191235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.191510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.192078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.193474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:12.202920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:12.204046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:12.204902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 460/927] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/services/object-store => generated 93 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:12.205702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.232382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.251712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55d04beb-f0c6-4ed0-b621-e188c3652fc8 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.285505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.286058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.286335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.286606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.297370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.298821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.305800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a1167a-ce73-49bb-bc32-e9673acfb624 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-7eefe88d-bc81-4090-8b3a-ca18c113b1a6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:12.313159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:12.314138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:12.315052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:12.315052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:12.315052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:12.315052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 468/928] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:12.315488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0904365-24bb-4575-92c2-9de0c938b25b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.315702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0904365-24bb-4575-92c2-9de0c938b25b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.315929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0904365-24bb-4575-92c2-9de0c938b25b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.317130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 469/929] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:12 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:12.317813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0904365-24bb-4575-92c2-9de0c938b25b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.321333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.347909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.348546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.348927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.349206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.350361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.350889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.352409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.361664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 461/930] 10.222.0.64 () {56 vars in 1114 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:12.362456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.420458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0904365-24bb-4575-92c2-9de0c938b25b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.449838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.455143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.455744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.456018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.456286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.456899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.457196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.458619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec0459bf-0715-4bc2-a862-85650768ca50 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.460538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e96d489-a872-4dfe-bf6b-f3b84e8b435a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.465408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 470/931] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:12.465408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.465408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.465651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.483445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:12.484639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:12.496417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.497017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.497301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.497561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.498190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.498486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.499864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.511625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 462/932] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:12.512398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.546587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.552352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 471/933] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:12 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:12.553385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.615416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3db1807-a137-4403-b932-5ef9a1bcaa4e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.649768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.650652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.651182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.651496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63741c4-39f2-4776-8f17-f667b138b7c6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.652138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.652984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.654605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.676838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.685158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.685703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.685978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.686244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.686895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.687194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.688547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.693275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 472/934] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:12.693931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.760381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 463/935] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:18:12 2022] POST /identity/v3/endpoints => generated 346 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:12.760381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.798732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da08ad19-cf6e-4284-9ea5-1ce9c698619b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.844123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.845053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.845334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.845604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.855086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.856566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:12.871229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:12.872365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 473/936] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:12.873255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.879355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.893020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b689c42-90e3-48ed-857a-a2815e211054 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:12.927552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:12.938132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:12.938748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:12.939097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:12.939323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:12.940026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] RBAC: Adding query filter params (interface=public, service_id=9d25ae04526245c190bda0150462969d, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:12.940647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:12.941818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:12.948891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 464/937] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/endpoints?service_id=9d25ae04526245c190bda0150462969d&interface=public®ion_id=RegionOne => generated 194 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:12.949421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:12.985728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fc51e96-3dd0-43e5-b7a1-bec421497e50 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.024701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.025371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.025599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.025866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.026434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.027819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:13.042545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:13.043745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:13.045515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 474/938] 10.222.0.64 () {56 vars in 1132 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:13.046334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:13.071634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6087b9-a35e-4eb0-bd04-7521306fbd83 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.112894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.113804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.114085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.114356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.115092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:13.115547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.116889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.129210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 465/939] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:13.136613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:13.152342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55bd5bd9-1472-457a-9b67-68bf2a8f34cf None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.156670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:13.163870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:13.187317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.187923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.188200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.188475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.189116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.190477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.197036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 475/940] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:18:13 2022] GET /identity/v3/services => generated 1497 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:18:13.197799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:13.285199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d7d37f9-7429-4517-bdb5-768777246178 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.324439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.325151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.325264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.325507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.326401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:13.326762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.328126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.329521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8a500d-04b1-44f7-b48a-6aeeaa997bc0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.338726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 466/941] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:13 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:13.340881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:13.383630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5893740d-9998-4bdb-99a2-cce5a4308d2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.384853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893740d-9998-4bdb-99a2-cce5a4308d2b None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.385285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893740d-9998-4bdb-99a2-cce5a4308d2b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.385738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5893740d-9998-4bdb-99a2-cce5a4308d2b None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/b821384a2058464892f0a0f37f34b914/roles/b0033600d8264a8d932bf36ce195ab0c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.389154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5893740d-9998-4bdb-99a2-cce5a4308d2b None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=b821384a2058464892f0a0f37f34b914, role_id=b0033600d8264a8d932bf36ce195ab0c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.390859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5893740d-9998-4bdb-99a2-cce5a4308d2b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.438434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:13.460701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d32d032-eacc-4fb9-a4ae-871188f95b7b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.497171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.498166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.498645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.499149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.500003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:13.500531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.502245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.508057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 467/942] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:13 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:13.513227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:13.523492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 476/943] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:18:13 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/b821384a2058464892f0a0f37f34b914/roles/b0033600d8264a8d932bf36ce195ab0c => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:18:13.643104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5961c6-688e-47dd-9b8f-84609fb36b76 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:13.681096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c6fda2-7ea2-4151-adf2-5be4dc57d96a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:13.681728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6fda2-7ea2-4151-adf2-5be4dc57d96a None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:13.682016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6fda2-7ea2-4151-adf2-5be4dc57d96a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:13.682283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c6fda2-7ea2-4151-adf2-5be4dc57d96a None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:13.682995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c6fda2-7ea2-4151-adf2-5be4dc57d96a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:13.684406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c6fda2-7ea2-4151-adf2-5be4dc57d96a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:13.756796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:13.763258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 468/944] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:18:13 2022] POST /identity/v3/endpoints => generated 347 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:15.008322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4af151d-1ae2-4491-a03f-ccb23138ae23 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.008881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4af151d-1ae2-4491-a03f-ccb23138ae23 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.008881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4af151d-1ae2-4491-a03f-ccb23138ae23 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.010422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 477/945] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:15.015950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de16e881-48a3-4797-92aa-e21b8761fdfc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.016347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de16e881-48a3-4797-92aa-e21b8761fdfc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.016534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de16e881-48a3-4797-92aa-e21b8761fdfc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.035123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-de16e881-48a3-4797-92aa-e21b8761fdfc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:15.036296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-de16e881-48a3-4797-92aa-e21b8761fdfc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:15.116813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 469/946] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:15.389397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4af151d-1ae2-4491-a03f-ccb23138ae23 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:15.429003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:15.496714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4af151d-1ae2-4491-a03f-ccb23138ae23 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:15.523234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018ea26-f5ec-4287-9f98-bfdb3b19e92d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.523565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018ea26-f5ec-4287-9f98-bfdb3b19e92d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.523820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c018ea26-f5ec-4287-9f98-bfdb3b19e92d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.526167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 470/947] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:15 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:15.531301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.531566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.531833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.533714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:15.537374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.537374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.537374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.537374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:15.537374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:15.540482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING keystone.server.flask.common [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 27 16:18:15.555629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:15.556980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:15.595908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.sql.core [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 27 16:18:15.595908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 27 16:18:15.595908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [parameters: {'id': '9372741129b549b88c71c93f4418b8d2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 27 16:18:15.595908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81612) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 9372741129b549b88c71c93f4418b8d2.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 9372741129b549b88c71c93f4418b8d2. Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application conn.query(q) Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 16:18:15.604463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result.read() Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Sep 27 16:18:15.605291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return project_ref.to_dict() Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application next(self.gen) Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application yield resource Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application next(self.gen) Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.session.rollback() Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.force_reraise() Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise self.value Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application session.commit() Sep 27 16:18:15.606721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._prepare_impl() Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.session.flush() Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._flush(objects) Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application compat.raise_( Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application flush_context.execute() Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 27 16:18:15.607837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rec.execute(self) Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application c = connection._execute_20( Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 27 16:18:15.608934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application conn.query(q) Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application result.read() Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 16:18:15.609914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [parameters: {'id': '9372741129b549b88c71c93f4418b8d2', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:15.611286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:15.612959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.Conflict( Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id 9372741129b549b88c71c93f4418b8d2. Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 478/948] 10.222.0.64 () {60 vars in 1140 bytes} [Tue Sep 27 16:18:15 2022] POST /identity/v3/projects => generated 262 bytes in 217 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:15.616123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:15.648727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 471/949] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:15.722110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:15.732888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b169190e-2065-4652-b16e-94c9859b955a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:15.768362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:15.768999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.769337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.769611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.777315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:15.779075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-0c95a170-b26a-4352-b61d-beee2615c9eb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:15.791620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:15.792727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:15.793853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:15.796197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 479/950] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:15 2022] GET /identity/v3/projects/service => generated 88 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:15.801763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:15.816697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7c800-9dae-4991-a44a-a893d9d9e91d None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.817030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7c800-9dae-4991-a44a-a893d9d9e91d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.817257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cf7c800-9dae-4991-a44a-a893d9d9e91d None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.822809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 480/951] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:15 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:15.829056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.829466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.829738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.854332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:15.855941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:15.860961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:15.912807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0247ab74-0b7c-4d9c-8b7a-8e793c70e524 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:15.927374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 481/952] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:15 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:15.944171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:15.955721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:15.956383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:15.956663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:15.956931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:15.958043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:15.958349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:15.960409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69767885-b3b3-482a-8433-7f167b3f8919 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:15.990905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 472/953] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:15 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:16.044578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:16.067368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75a3dec2-9bdb-4ec8-8b60-9d183e1911f3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.107407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.108420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.108911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.109443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.110275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.111960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:16.121354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7b7f9-f437-4f63-9bc4-47d6dc709ee3 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.121899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7b7f9-f437-4f63-9bc4-47d6dc709ee3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.121899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ec7b7f9-f437-4f63-9bc4-47d6dc709ee3 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.123588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 473/954] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:16.124017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:16.125102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:16.126084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:16.128949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 482/955] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:15 2022] GET /identity/v3/services/object-store => generated 93 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:16.129071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.129260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.133120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.136225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.153069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:16.154287 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:16.241008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 474/956] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:16 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:16.242491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.258902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7519c15c-31c1-44af-b490-cdaae34d18cc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.299282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:16.301398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.302007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.303210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.303398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.304101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:16.304405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.306282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:16.312689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 483/957] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:16.319062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.332012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:16.377704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddb81ffe-880c-402c-977c-af2f41a9da60 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.415418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.416145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.416425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.416794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.427892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:16.433069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:16.434094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:16.434976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 475/958] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:16.440969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.457533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9aa86393-4f80-441d-a0dc-312316791d85 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.501811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.502424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.502702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.503026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.503814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:16.504467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.506170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:16.512375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 484/959] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:16.518313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.559629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bddd7283-1028-4466-9f20-3ad7ad1a36ce None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.561993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:16.595602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.596154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.596818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.597945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.598604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:16.599095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.601057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:16.607509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:16.610158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 476/960] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:16.610915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.644986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b295ab-969e-4480-858e-64f0a22b2b56 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.680262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.681211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.681687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.682152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.683099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.684835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:16.722887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573af5a5-bf51-475c-89f5-ce796dd80473 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.757188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.758185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.758650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.759148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] PATH_INFO: `/v3/services/compute` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.760187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.762563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:16.774665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:16.775774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:16.776647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:16.777889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 477/961] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/services/compute => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:16.778906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.799159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 485/962] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:18:16 2022] POST /identity/v3/endpoints => generated 349 bytes in 281 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:16.800384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:16.814527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:16.917106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-602bf049-b897-461d-b6f3-8d35bfdd9e1c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.969743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6461332-7603-4ac4-903f-a64ade8467f5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:16.977419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:16.978144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:16.978533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:16.979474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:16.995593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:16.997896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.001413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:17.014687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:17.015802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:17.016728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:17.016728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:17.016728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:17.017404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 478/963] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 239 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:17.017982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:17.018763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:17.019224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:17.019708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:17.020315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] RBAC: Adding query filter params (name=compute) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:17.020623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:17.022563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.025106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:17.028243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 486/964] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:16 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:17.041752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:17.116202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:17.139091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1da135-3800-43bd-afc3-e2fe1a91523f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:17.177293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:17.178492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:17.178756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:17.179401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:17.180587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:17.180968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:17.182708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.183630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0209c2e-aa97-4212-9a16-c7df974f597a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:17.202828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 479/965] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:17.205802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:17.230582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:17.231227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:17.231509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:17.231785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:17.232420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] RBAC: Adding query filter params (type=compute) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:17.232723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:17.234150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.246812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 487/966] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:17 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:17.337359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edfa57d5-0b1f-42c6-8bf8-c77b11f1f3f3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:17.378253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:17.378939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:17.379221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:17.379497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:17.388385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:17.389857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.392332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:17.403943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:17.417275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:17.418308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 480/967] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:18:17 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:17.418945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:17.479570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:17.524516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe536f2f-bb40-46e1-83a4-d62e198b828d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:17.561360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:17.562339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:17.562868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:17.563472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:17.564402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:17.564961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:17.566746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.586756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 488/968] 10.222.0.64 () {56 vars in 1149 bytes} [Tue Sep 27 16:18:17 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:17.596088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:17.703140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cef82886-19d3-495d-848e-6363a6ccdd3f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:17.738600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:17.739157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:17.739434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:17.739702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:17.811862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:17.815551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:17.817017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:17.920921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 481/969] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:17 2022] GET /identity/v3/auth/tokens => generated 1474 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:17.926597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:18.031298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea527268-ec49-4f9f-905f-d8edb659b140 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:18.043360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:18.068607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:18.069814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.070301 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.070770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.082852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:18.084672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:18.104844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:18.105834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:18.106730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:18.106730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:18.106730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:18.106730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 489/970] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:18.113084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:18.216655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:18.221727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfa1ba9f-de0b-455b-97e3-c7f446ac6a8a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:18.262648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:18.263742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.264252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.264737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.265608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:18.266118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:18.267958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da4cb943-40cd-4225-b04b-a93afbd53e56 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:18.281535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 482/971] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:18.288317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:18.360442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dbf60c-be11-4229-9b96-62ed25802862 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.360729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dbf60c-be11-4229-9b96-62ed25802862 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.361011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3dbf60c-be11-4229-9b96-62ed25802862 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.362483 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 483/972] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:18.368061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.368368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.368617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.387171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:18.388384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:18.413391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2144794-26c7-4355-950a-98789ee5bf56 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:18.456413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:18.458024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.458547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.459563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.470809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:18.470995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 484/973] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:18 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:18.472660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:18.485377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:18.486497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:18.487637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:18.489703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 490/974] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:18 2022] GET /identity/v3/projects/service => generated 88 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:18.498347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:18.590350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:18.603179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0489dcbc-0b07-4251-9b6c-fd6577dd4037 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:18.645544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:18.646436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.646769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.647069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.647764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:18.648069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:18.649626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:18.662019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 485/975] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:18 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:18.667800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:18.696888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:18.775110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd7ec536-a852-4d96-8814-4316ee10e1af None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:18.807017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:18.809484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:18.810212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.810508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.810949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.811726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] RBAC: Adding query filter params (role.id=b0033600d8264a8d932bf36ce195ab0c, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=b821384a2058464892f0a0f37f34b914) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:18.812033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:18.813496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:18.820630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 491/976] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:18 2022] GET /identity/v3/role_assignments?user.id=b821384a2058464892f0a0f37f34b914&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=b0033600d8264a8d932bf36ce195ab0c => generated 608 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:18.924365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c0693a-cef8-4246-8ca8-6db01fc9faaa None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:18.963332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:18.964056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:18.964571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:18.965060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:18.966256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:18.967979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:18.971629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING keystone.server.flask.common [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Sep 27 16:18:19.018481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.sql.core [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 27 16:18:19.018481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 27 16:18:19.018481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [parameters: {'user_id': 'a72fb3ad0a98445f8ecee4988403a99d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 27 16:18:19.018481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=81613) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-07b84392-41ee-4c8a-8bd0-2358950bf589 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application conn.query(q) Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result.read() Sep 27 16:18:19.035491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return method(*args, **kwargs) Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 16:18:19.036419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application next(self.gen) Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application yield resource Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application next(self.gen) Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.session.rollback() Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.force_reraise() Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise self.value Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application session.commit() Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Sep 27 16:18:19.038988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._prepare_impl() Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.session.flush() Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._flush(objects) Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application compat.raise_( Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application flush_context.execute() Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rec.execute(self) Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Sep 27 16:18:19.041153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application _emit_insert_statements( Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result = connection._execute_20( Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1705, in _execute_20 Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 333, in _execute_on_connection Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ret = self._execute_context( Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1943, in _execute_context Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2122, in _handle_dbapi_exception Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Sep 27 16:18:19.043327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self.dialect.do_execute( Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result = self._query(query) Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application conn.query(q) Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application result.read() Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Sep 27 16:18:19.044212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application packet.raise_for_error() Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [parameters: {'user_id': 'a72fb3ad0a98445f8ecee4988403a99d', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:19.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Sep 27 16:18:19.045919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.047041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 486/977] 10.222.0.64 () {60 vars in 1131 bytes} [Tue Sep 27 16:18:18 2022] POST /identity/v3/users => generated 160 bytes in 231 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:19.047446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:19.120627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f77224-cef3-4805-95f5-25d946978c5e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.121141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f77224-cef3-4805-95f5-25d946978c5e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.121630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f77224-cef3-4805-95f5-25d946978c5e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.126450 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 487/978] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:19 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:19.132762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.133343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.133818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.141150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:19.156272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:19.158050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88653cb4-1c44-4f2c-bb25-d13f82cf4b21 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:19.158401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:19.161743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:19.200345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:19.202040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.202326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.202596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.212914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:19.214645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:19.228799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:19.229808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Sep 27 16:18:19.230761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:19.237553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 492/979] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:18:19 2022] GET /identity/v3/users/cinder => generated 84 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:19.238315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:19.258837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 488/980] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:19.350423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb2a7ee-b569-4980-bd2d-617347a9e5a5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:19.386921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:19.387535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.387816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.388090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.388784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] RBAC: Adding query filter params (name=cinder) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:19.389086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:19.390523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6dc95dc-668e-4ee6-868f-c42f5af2cf1e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:19.427542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 493/981] 10.222.0.64 () {56 vars in 1101 bytes} [Tue Sep 27 16:18:19 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:19.535964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:19.654090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e91bc838-540d-4173-94ba-cc3c11476cf8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:19.668350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8853a2-acae-430d-8ce6-2a1a0fbbf5fe None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.668645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8853a2-acae-430d-8ce6-2a1a0fbbf5fe None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.668898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8853a2-acae-430d-8ce6-2a1a0fbbf5fe None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.670268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 494/982] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:19.675635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.675919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.676186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.685996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:19.694756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:19.695442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.695715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.697203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.697203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:19.697913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:19.698048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:19.699333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:19.708949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:19.710146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:19.711039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 489/983] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:19 2022] GET /identity/v3/services/object-store => generated 93 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:19.716436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:19.721195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:19.798437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 495/984] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:19 2022] POST /identity/v3/auth/tokens => generated 1474 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:19.831665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c895d8-9f66-45c6-8fab-1eb1a1f533bc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:19.868760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:19.869371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:19.869652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:19.869924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:19.870550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:19.870945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:19.872324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:19.882162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 490/985] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:19 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:19.883096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:19.983292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee3c70e-ca8e-4538-86dc-7c91390ebfdf None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:20.015696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:20.016293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:20.016566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:20.016887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:20.017503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:20.017798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:20.019171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:20.020584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:20.025531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 496/986] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:19 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:20.031406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:20.069884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:20.099872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:20.137622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2422247f-30d0-437d-b75e-2b1766449a0e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:20.172490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:20.173105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:20.173383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:20.173656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:20.174363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] RBAC: Adding query filter params (interface=admin, service_id=9d25ae04526245c190bda0150462969d, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:20.174671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:20.176088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:20.181237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b40df80b-b5e3-4c2b-a879-4e92654be4c1 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:20.182049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 491/987] 10.222.0.64 () {56 vars in 1249 bytes} [Tue Sep 27 16:18:20 2022] GET /identity/v3/endpoints?service_id=9d25ae04526245c190bda0150462969d&interface=admin®ion_id=RegionOne => generated 193 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:20.187642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:20.222877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-216a0ac8-a55b-4658-a35c-43109b91e952 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:20.223491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a0ac8-a55b-4658-a35c-43109b91e952 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:20.223770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a0ac8-a55b-4658-a35c-43109b91e952 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:20.224068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-216a0ac8-a55b-4658-a35c-43109b91e952 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:20.224728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216a0ac8-a55b-4658-a35c-43109b91e952 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:20.226085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-216a0ac8-a55b-4658-a35c-43109b91e952 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:20.290438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34053940-cd43-44bf-a970-4ab2620ca05d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:20.312252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 497/988] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:20 2022] POST /identity/v3/services => generated 242 bytes in 243 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:20.331010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:20.344741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182ad1a7-bee6-45d8-9c88-73a831e14006 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:20.345385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ad1a7-bee6-45d8-9c88-73a831e14006 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:20.345665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ad1a7-bee6-45d8-9c88-73a831e14006 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:20.346096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182ad1a7-bee6-45d8-9c88-73a831e14006 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:20.346718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182ad1a7-bee6-45d8-9c88-73a831e14006 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:20.348407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182ad1a7-bee6-45d8-9c88-73a831e14006 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:20.362874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 492/989] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:18:20 2022] GET /identity/v3/services => generated 1727 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:18:21.134479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7ac1b-d3c1-4c23-9b78-013bd7665122 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:21.135035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7ac1b-d3c1-4c23-9b78-013bd7665122 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:21.135035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b7ac1b-d3c1-4c23-9b78-013bd7665122 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:21.136392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 498/990] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:21.141785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:21.142076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:21.142356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:21.160439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:21.162256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:21.231419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 493/991] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:21 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:21.496434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7ac1b-d3c1-4c23-9b78-013bd7665122 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:21.561530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:21.606446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b7ac1b-d3c1-4c23-9b78-013bd7665122 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:21.642684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:21.643333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:21.643608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:21.643870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] PATH_INFO: `/v3/services/image` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:21.644439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:21.645790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:21.655017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:21.656037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:21.657443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 499/992] 10.222.0.64 () {56 vars in 1105 bytes} [Tue Sep 27 16:18:21 2022] GET /identity/v3/services/image => generated 86 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:21.662009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:21.764859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4fb97f-44eb-41d2-bd54-3a18abfefae8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:21.789947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:21.801095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:21.801706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:21.801984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:21.802435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:21.803181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] RBAC: Adding query filter params (name=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:21.803489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:21.804845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fecc6dd5-606d-48d8-8716-3f07c47f2ce7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:21.809637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 494/993] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:21 2022] GET /identity/v3/services?name=image => generated 122 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:21.815116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:21.915672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-083bb3da-516c-4d93-9d27-203647193334 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:21.937108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:21.948659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:21.950434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:21.950434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:21.950434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:21.950434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] RBAC: Adding query filter params (type=image) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:21.951013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:21.952299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-225b51ca-61a2-4ef2-b972-5355ffac8480 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:21.956847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 500/994] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:21 2022] GET /identity/v3/services?type=image => generated 122 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:22.561079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac49cfc-e916-4b9b-8bfb-25fa911420e5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:22.561360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac49cfc-e916-4b9b-8bfb-25fa911420e5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:22.561830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cac49cfc-e916-4b9b-8bfb-25fa911420e5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:22.563372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 495/995] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:22.568796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:22.569241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:22.569332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:22.587966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:22.589152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:22.638160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631f8f1-e102-4955-a2b6-cda0ebb8a292 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:22.638160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631f8f1-e102-4955-a2b6-cda0ebb8a292 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:22.638160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c631f8f1-e102-4955-a2b6-cda0ebb8a292 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:22.638845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 496/996] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:22.644132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:22.644417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:22.644717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:22.657541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 501/997] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:22 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:22.663693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:22.664922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:22.733683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 497/998] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:22 2022] POST /identity/v3/auth/tokens => generated 1570 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:22.925904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.000772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.026796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba5262f-fcc1-4e41-8bbd-fb2409ff2e85 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.044106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:23.063823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.064720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.065003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.065271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] PATH_INFO: `/v3/services/compute` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.065843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.067236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:23.076286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:23.077329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:23.078207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 502/999] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:22 2022] GET /identity/v3/services/compute => generated 88 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:23.083148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.111177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63830864-20c4-4224-85e4-913fa6d4e7d3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.128623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:23.147331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.147966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.148244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.148515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] PATH_INFO: `/v3/services/object-store` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.149218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.150600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:23.159364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:23.160352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:23.163861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 498/1000] 10.222.0.64 () {56 vars in 1126 bytes} [Tue Sep 27 16:18:22 2022] GET /identity/v3/services/object-store => generated 93 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:23.166407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.193910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-215071ca-753b-4e5b-8125-cda2a5a46bb4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.230030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.230632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.230944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.231215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.231835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:23.232132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.233489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.238540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 503/1001] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:23 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:23.244806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.271125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56575a43-db41-418d-93da-4a5d2d53e80a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.291010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:23.312090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.312870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.313151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.313424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.314083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:23.314389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.316796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.330256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 499/1002] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:23 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:23.330256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.361940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7891bd2c-959d-4565-bcb7-8959e323ee00 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.388097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:23.407227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.408264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.408789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.409252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.410452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:23.411029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.415033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.429515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 504/1003] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:23 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:23.431390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.449504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f62a6d-84ae-4d25-ad62-383020a8aa34 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.484706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.485317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.485590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.485860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.486481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] RBAC: Adding query filter params (type=object-store) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:23.486810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.488158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.493158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 500/1004] 10.222.0.64 () {56 vars in 1122 bytes} [Tue Sep 27 16:18:23 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:23.499099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.551172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20805e57-5c95-4b91-bec9-90e1ab420c95 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.563163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:23.592498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.593167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.593360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.593628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] PATH_INFO: `/v3/endpoints` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.594320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] RBAC: Adding query filter params (interface=public, service_id=996623d650734ecfa2a9939290cba15d, region_id=RegionOne) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:23.594626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.596009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.602371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 505/1005] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:18:23 2022] GET /identity/v3/endpoints?service_id=996623d650734ecfa2a9939290cba15d&interface=public®ion_id=RegionOne => generated 194 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:23.603468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d30e2690-9caa-4fd2-8336-2f56e8b6b33c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.607597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:23.612913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:23.637852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-414d27e9-fdfb-440f-869f-8418cefb34b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.638457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d27e9-fdfb-440f-869f-8418cefb34b5 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.638738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d27e9-fdfb-440f-869f-8418cefb34b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.639037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-414d27e9-fdfb-440f-869f-8418cefb34b5 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.639693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d27e9-fdfb-440f-869f-8418cefb34b5 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.641055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-414d27e9-fdfb-440f-869f-8418cefb34b5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.716921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e98b408-241e-4811-8383-727df69ad1de None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:23.721725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 501/1006] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:18:23 2022] POST /identity/v3/endpoints => generated 325 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:23.761648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0ec748-cf94-4596-a671-e3bade61a423 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:23.762277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0ec748-cf94-4596-a671-e3bade61a423 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:23.762556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0ec748-cf94-4596-a671-e3bade61a423 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:23.762860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0ec748-cf94-4596-a671-e3bade61a423 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:23.763529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0ec748-cf94-4596-a671-e3bade61a423 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:23.764909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0ec748-cf94-4596-a671-e3bade61a423 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:23.772849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 506/1007] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:18:23 2022] GET /identity/v3/services => generated 1727 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:18:24.202489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849b8113-0b00-4635-8e3a-48098f922d71 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:24.202889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849b8113-0b00-4635-8e3a-48098f922d71 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:24.203075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849b8113-0b00-4635-8e3a-48098f922d71 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:24.204550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 502/1008] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:24.209915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e74baa-7106-4587-8e49-482820a2efb5 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:24.210198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e74baa-7106-4587-8e49-482820a2efb5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:24.210468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e74baa-7106-4587-8e49-482820a2efb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:24.229379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-89e74baa-7106-4587-8e49-482820a2efb5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:24.230572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-89e74baa-7106-4587-8e49-482820a2efb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:24.257192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:24.298196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 507/1009] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:24 2022] POST /identity/v3/auth/tokens => generated 1717 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:24.571840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849b8113-0b00-4635-8e3a-48098f922d71 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:24.678229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849b8113-0b00-4635-8e3a-48098f922d71 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:24.715500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d776f5ee-0029-4425-b1ba-a4850a65d293 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:24.716152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d776f5ee-0029-4425-b1ba-a4850a65d293 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:24.716433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d776f5ee-0029-4425-b1ba-a4850a65d293 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:24.716701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d776f5ee-0029-4425-b1ba-a4850a65d293 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:24.717343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d776f5ee-0029-4425-b1ba-a4850a65d293 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:24.718763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d776f5ee-0029-4425-b1ba-a4850a65d293 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:24.828607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 503/1010] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:24 2022] POST /identity/v3/services => generated 242 bytes in 253 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:25.911997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbe06d-a075-4118-bfd3-dfcc5fea15a7 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:25.912523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbe06d-a075-4118-bfd3-dfcc5fea15a7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:25.912523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58dbe06d-a075-4118-bfd3-dfcc5fea15a7 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:25.914019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 508/1011] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:25.919209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:25.919610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:25.919770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:25.937933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:25.939141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:25.963988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:25.992133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2396bd4b-7ac9-4c10-9a22-e698c559182f None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:25.992480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2396bd4b-7ac9-4c10-9a22-e698c559182f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:25.992753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2396bd4b-7ac9-4c10-9a22-e698c559182f None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:25.994138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 509/1012] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:25.999934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.000203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.000482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.016017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 504/1013] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:25 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:26.023774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:26.025019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:26.095509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 510/1014] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:25 2022] POST /identity/v3/auth/tokens => generated 1813 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:26.282249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:26.361779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:26.390743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a7888e-e88b-4889-aba5-01bc6def2819 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:26.428303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:26.428941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.429219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.429488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.430579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:26.432052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:26.433977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 505/1015] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:26 2022] GET /identity/v3/domains/default => generated 199 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:26.439279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:26.459437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:26.465315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:26.469473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f768982b-16a0-41f7-88b7-15756064eaa1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:26.506133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:26.507169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.507661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.508016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] PATH_INFO: `/v3/services/compute` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.508598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:26.510002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:26.519123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:26.520643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:26.521873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 511/1016] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:26 2022] GET /identity/v3/services/compute => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:26.526280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:26.547592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab896c4e-db9d-4a0a-9603-02152ab8a6c2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:26.582919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:26.583621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.584076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.584367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.585019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:26.586464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:26.632546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8bc2ba1-b9f9-4f79-bb40-f5c46b9709fc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:26.667096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:26.667700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.667978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.668285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.668906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] RBAC: Adding query filter params (name=compute) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:26.669205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:26.670559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:26.675569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 512/1017] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:26 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:26.681728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:26.704540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:26.714824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 506/1018] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:26 2022] POST /identity/v3/projects => generated 311 bytes in 268 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:26.792105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6e093f5-8778-4f71-a0a5-af291f34ad43 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:26.828893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:26.829422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.829695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.829961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.830595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] RBAC: Adding query filter params (type=compute) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:26.830934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:26.832327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8717cd8d-862c-4b3d-a9d8-5062f0c7b794 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:26.843294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 513/1019] 10.222.0.64 () {56 vars in 1112 bytes} [Tue Sep 27 16:18:26 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:26.844519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:26.931543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:26.959666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f763b16-8332-4115-95d9-50c187cbdbd7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:26.995425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:26.996042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:26.996327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:26.996601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:26.997284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:26.998648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:27.033039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5381ed-356d-4ab7-bd33-6d66fd11fc8d None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:27.033555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5381ed-356d-4ab7-bd33-6d66fd11fc8d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:27.033555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e5381ed-356d-4ab7-bd33-6d66fd11fc8d None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:27.035101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 514/1020] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:27.039817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:27.040169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:27.040364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:27.058929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:27.060046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:27.087675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 507/1021] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:18:26 2022] POST /identity/v3/endpoints => generated 334 bytes in 237 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:27.138347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 515/1022] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:27 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:27.406650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:27.512774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-299ac3cd-bb9b-4119-a94c-96e287d0ff13 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:27.543529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:27.550043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:27.550687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:27.550998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:27.551278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] PATH_INFO: `/v3/services/image` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:27.551976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:27.553403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:27.562400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:27.563632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:27.565057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 508/1023] 10.222.0.64 () {56 vars in 1105 bytes} [Tue Sep 27 16:18:27 2022] GET /identity/v3/services/image => generated 86 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:27.569492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:27.581997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:27.673616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fe8751e-5b73-4d8f-89b3-45ada69a7f11 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:27.711752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:27.712379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:27.712657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:27.712929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:27.713569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] RBAC: Adding query filter params (name=image) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:27.713874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:27.715299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:27.726222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 516/1024] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:27 2022] GET /identity/v3/services?name=image => generated 122 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:27.727232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:27.826991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e04d015-f465-4cca-a1e3-98de73e119e4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:27.861697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:27.862308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:27.862586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:27.862883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:27.863507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] RBAC: Adding query filter params (type=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:27.863810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:27.865217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:27.874862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 509/1025] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:27 2022] GET /identity/v3/services?type=image => generated 350 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:27.875920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:27.953578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:27.979156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5e04069-260e-4338-a639-0d0eeb73f714 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:28.016741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:28.017355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:28.017635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:28.017907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] PATH_INFO: `/v3/endpoints` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:28.018609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] RBAC: Adding query filter params (interface=public, service_id=e6ce011d724f4203ab10b18b74e0a6f0, region_id=RegionOne) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:28.018952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:28.020386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b612cd-4018-466b-a94a-6abac225192b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:28.031086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 517/1026] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:18:27 2022] GET /identity/v3/endpoints?service_id=e6ce011d724f4203ab10b18b74e0a6f0&interface=public®ion_id=RegionOne => generated 194 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:28.032358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:28.097653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:28.140523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46738804-9dfe-4546-915d-3b365bcc9a13 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:28.174283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b354642-35e1-4dbf-8311-36b5801a1566 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:28.174908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b354642-35e1-4dbf-8311-36b5801a1566 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:28.175186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b354642-35e1-4dbf-8311-36b5801a1566 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:28.175453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b354642-35e1-4dbf-8311-36b5801a1566 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:28.176042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b354642-35e1-4dbf-8311-36b5801a1566 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:28.177688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b354642-35e1-4dbf-8311-36b5801a1566 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:28.190429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 510/1027] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:18:28 2022] GET /identity/v3/services => generated 1957 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:18:29.004222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db316eaa-7ab5-4c56-a502-f1af4dc2354b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.004768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db316eaa-7ab5-4c56-a502-f1af4dc2354b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.004768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db316eaa-7ab5-4c56-a502-f1af4dc2354b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.006712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 518/1028] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:29.012223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc39936a-2b50-4a5f-9bc4-43935f9a07f6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.012512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc39936a-2b50-4a5f-9bc4-43935f9a07f6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.012788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc39936a-2b50-4a5f-9bc4-43935f9a07f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.031082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-dc39936a-2b50-4a5f-9bc4-43935f9a07f6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:29.032282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dc39936a-2b50-4a5f-9bc4-43935f9a07f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:29.103846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 511/1029] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:29 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:29.373622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db316eaa-7ab5-4c56-a502-f1af4dc2354b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:29.383247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f82864-32e3-4ca9-9854-00fd76971dc2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.383545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f82864-32e3-4ca9-9854-00fd76971dc2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.383825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52f82864-32e3-4ca9-9854-00fd76971dc2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.385171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 512/1030] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:29.390140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.390423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.390701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.410166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:29.412045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:29.461885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:29.481930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 513/1031] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:29 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:29.486134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db316eaa-7ab5-4c56-a502-f1af4dc2354b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:29.522687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:29.523341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.523622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.523890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.524992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:29.526411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:29.528176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 519/1032] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:29 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:29.533615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:29.640066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72e2be35-7f61-41e2-ab9c-f37d4dc0db44 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:29.674919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:29.675526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.675805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.676085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.676761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:29.678650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:29.752661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:29.762356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:29.784710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 514/1033] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:18:29 2022] POST /identity/v3/users => generated 285 bytes in 252 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:29.873414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f6fb6b4-5b11-4ca6-8a80-45a1e674bf6f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:29.909798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:29.910368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:29.910648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:29.910951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:29.923939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:29.930126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:29.931200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:29.932775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 520/1034] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Sep 27 16:18:29 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:29.942965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:29.987914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:30.051569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05a9b228-b116-460a-b720-7b54854252f4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.089456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.090109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.090389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.090655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.091342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:30.091642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:30.093002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:30.106806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 515/1035] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:18:29 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:30.109461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.179352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:30.215340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-823e009d-9f18-43eb-bef7-6f7bb92d509a None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.250294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.251214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.251643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.252090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.265003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:30.266738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d4f79570-a44f-4ce7-af43-4e9b563f1564 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:30.282906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:30.284130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:30.285224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:30.285224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:30.285224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:30.287926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 521/1036] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:30.293685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.381883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0c6319-063c-4f15-be78-5fd87a727e27 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.382204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0c6319-063c-4f15-be78-5fd87a727e27 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.383527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0c6319-063c-4f15-be78-5fd87a727e27 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.384016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 522/1037] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:30.389023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.389304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.389571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.405124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:30.408287 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf488df-3d51-4e00-92b9-7209827124c2 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.409881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:30.411453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:30.446564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.447124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.447396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.447665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.448298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:30.448602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:30.450858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:30.462995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 516/1038] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:30.468341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.483896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 523/1039] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:30 2022] POST /identity/v3/auth/tokens => generated 1969 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:30.569027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3dede699-52d6-4583-8366-451a3362ab8e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.602183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.602834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.603110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.603374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.610894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:30.612328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:30.624365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:30.625314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:18:30.626266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:30.630882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 517/1040] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/users/nova => generated 82 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:30.632133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.683075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:30.734359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da26f8b0-05a8-4234-bb09-608fdc293244 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.760875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.773510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.774603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.774932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.775214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.776304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:30.776614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:30.778254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:30.787057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:30.801180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 524/1041] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:30.803170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.874591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37333c11-7a82-433b-80ee-b8a34fd1d09e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.911575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.912226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.912488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.912757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] PATH_INFO: `/v3/services/image` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:30.913325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:30.914686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:30.916298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e7fbd5b-e744-4bbc-bc8b-9e7403ac2682 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:30.923666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:30.924658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Sep 27 16:18:30.925589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:30.926509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:30.930092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 518/1042] 10.222.0.64 () {56 vars in 1105 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:30.931399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:30.956253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:30.957418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:30.958109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:30.958528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.024013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:31.031249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.033464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.040502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06ff1615-3b9b-4772-a92d-b9a16357078b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.077321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.077933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.078209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.078484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.079140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] RBAC: Adding query filter params (name=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:31.079446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.080816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.086454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 519/1043] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/services?name=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:31.091995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:31.160137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 525/1044] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:30 2022] GET /identity/v3/auth/tokens => generated 1969 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:31.165825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:31.174331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:31.204412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e013c766-6aa9-408e-b064-88e693871287 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.240429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.241030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.241307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.241575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.242193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] RBAC: Adding query filter params (type=image) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:31.242493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.243894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.253302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 520/1045] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:31 2022] GET /identity/v3/services?type=image => generated 350 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:31.254271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:31.269083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:31.274261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06cc5c8f-7244-4d57-9e3b-b884640b4969 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.309343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.309962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.310240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.310520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.320693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.322107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:31.341226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:31.342237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:31.344154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:31.344154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:31.344154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:31.344154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 526/1046] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:31.347976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:31.359876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4bb35f0-d497-4b7e-9ce0-b79211758c41 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.396769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.397463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.397743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.398015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.398698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.400292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.419852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:31.477722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d1a4c9d-27b1-4eec-ba5b-487428907c35 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.494830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 521/1047] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:18:31 2022] POST /identity/v3/endpoints => generated 327 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:31.521702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.522326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.522617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.522907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.523567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:31.523868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.525425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.544633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 527/1048] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:31.546067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:31.621922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:31.659899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7406c5-5441-422e-9033-d20b033ad084 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.698186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.698863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.699205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.699481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.708639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.709555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:31.725404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:31.726522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:31.727638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 522/1049] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:31 2022] GET /identity/v3/projects/service => generated 88 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:31.733510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:31.841698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a6e20a9-5f3a-47c2-a7b0-786b33376de4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:31.855650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:31.875443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:31.876048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:31.876325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:31.876595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:31.877279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:31.877581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:31.879106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c8abd49-4bfa-4d7d-957d-a2fb6ce23d85 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:31.892112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 528/1050] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:31 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:31.898690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:32.004393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0537df6d-6d39-43f0-af93-641e26d46773 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:32.031543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:32.039695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:32.046457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:32.046457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:32.046457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:32.046457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] RBAC: Adding query filter params (role.id=b0033600d8264a8d932bf36ce195ab0c, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=ab556e3087544c42929aab1b35d145a4) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:32.046457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:32.046457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d14dbb-b7c8-4f81-864b-72f80c4b054e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:32.055021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 523/1051] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:31 2022] GET /identity/v3/role_assignments?user.id=ab556e3087544c42929aab1b35d145a4&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=b0033600d8264a8d932bf36ce195ab0c => generated 259 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:32.087712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0c9e28-2760-4b33-8043-16c8ba7de5e0 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:32.088078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0c9e28-2760-4b33-8043-16c8ba7de5e0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:32.088463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa0c9e28-2760-4b33-8043-16c8ba7de5e0 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:32.089953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 529/1052] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:32.096000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:32.096259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:32.096472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:32.116861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:32.118092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:32.196285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 524/1053] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:32 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:32.465549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0c9e28-2760-4b33-8043-16c8ba7de5e0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:32.573151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa0c9e28-2760-4b33-8043-16c8ba7de5e0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:32.607152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:32.607781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:32.608058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:32.608326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:32.620162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:32.624936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:32.625866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:18:32.626765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:32.631206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 530/1054] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:18:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:32.632228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:32.732094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be43fa5e-3f35-4fc5-98f6-79b5dfacb1d6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:32.757453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:32.768952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:32.770355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:32.770355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:32.770355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:32.771052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:32.771359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:32.772724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:32.785997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 525/1055] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:18:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:32.787127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:32.823451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:32.888262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a3ec5af-a77e-44e5-b9dd-ce7c9e13ebbc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:32.922848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:32.923481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:32.923752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:32.924017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] PATH_INFO: `/v3/users/a66f2ab311a145349d77271a3e85ec21` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:32.938237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] RBAC: Authorizing `identity:get_user(user_id=a66f2ab311a145349d77271a3e85ec21)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:32.939683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:32.941471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 531/1056] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:32 2022] GET /identity/v3/users/a66f2ab311a145349d77271a3e85ec21 => generated 285 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:32.947661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:33.045750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f274be4a-fdda-49fe-8c11-2e0ee0ee75a1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:33.078496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:33.079119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:33.079396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:33.079661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:33.132747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:33.148981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:33.150442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:33.251184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 526/1057] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:32 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:33.257388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:33.356371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:33.365827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f84e0a4-b9a7-4969-bcdf-cfe7b5e38eae None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:33.401192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3928b52-3b1a-42f8-8cb2-166b08ba06fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:33.401834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3928b52-3b1a-42f8-8cb2-166b08ba06fa None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:33.402109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3928b52-3b1a-42f8-8cb2-166b08ba06fa None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:33.402380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3928b52-3b1a-42f8-8cb2-166b08ba06fa None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:33.403508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3928b52-3b1a-42f8-8cb2-166b08ba06fa None admin] RBAC: Authorizing `identity:get_project(project_id=dfce272ab2364d568042aaebcf0102e8)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:33.404929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3928b52-3b1a-42f8-8cb2-166b08ba06fa None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:33.410823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 532/1058] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:18:33 2022] GET /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:33.413177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:33.499051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:33.515862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9eb516c-aae5-4c8c-bed6-fa5203a9e9f5 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:33.550375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:33.551218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:33.552183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:33.552654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:33.553505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=dfce272ab2364d568042aaebcf0102e8, user.id=a66f2ab311a145349d77271a3e85ec21) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:33.553970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:33.555624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3828a6eb-1a91-4667-8f74-a9b9d883aa51 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:33.561170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 527/1059] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:33 2022] GET /identity/v3/role_assignments?user.id=a66f2ab311a145349d77271a3e85ec21&scope.project.id=dfce272ab2364d568042aaebcf0102e8&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:33.769924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0480738-00a6-4188-8d5f-d97cea765d95 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:33.771017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0480738-00a6-4188-8d5f-d97cea765d95 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:33.771575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0480738-00a6-4188-8d5f-d97cea765d95 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:33.773107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 533/1060] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:33 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:33.778229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e89e-c1b6-4364-b957-7d8a22cd2719 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:33.778652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e89e-c1b6-4364-b957-7d8a22cd2719 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:33.778843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8681e89e-c1b6-4364-b957-7d8a22cd2719 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:33.797681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8681e89e-c1b6-4364-b957-7d8a22cd2719 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:33.798909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8681e89e-c1b6-4364-b957-7d8a22cd2719 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:33.869676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 528/1061] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:33 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:34.147680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0480738-00a6-4188-8d5f-d97cea765d95 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:34.212641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5473e7ca-70cf-4b3b-b029-927740e5ae33 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.212984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5473e7ca-70cf-4b3b-b029-927740e5ae33 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.213198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5473e7ca-70cf-4b3b-b029-927740e5ae33 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.214538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 529/1062] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:34.219262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.219549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.219825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.237504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:34.238690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:34.253041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0480738-00a6-4188-8d5f-d97cea765d95 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:34.290560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:34.291229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.291507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.291779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.310554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 530/1063] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:34 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:34.321938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:34.368915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:34.370346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:34.474894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 534/1064] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:34 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:34.477567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:34.488666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:34.577568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d09097d-a33b-45ee-b5e2-01fcd3ebdded None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:34.590929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:34.616973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:34.617600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.617879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.618150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.628493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:34.630102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:34.644625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:34.645624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:34.646860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:34.646860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:34.646860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:34.650730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 531/1065] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:34.651461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:34.654680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:34.696586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9ff8215-fa9c-4cbd-88aa-58781f6b57b9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:34.715008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:34.735061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:34.736012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.736551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.737198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.749604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:34.755708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:34.756681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:34.757543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:34.758187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ffaa6c-2183-48fb-9064-0278ec5cbc75 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:34.759789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 535/1066] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Sep 27 16:18:34 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:34.765307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:34.801221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:34.802598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.802879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.803326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.804223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:34.804352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:34.805648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:34.818352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 532/1067] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:34.878574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ac6a8c6-4995-47d3-88d4-22620b6e53f4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:34.902951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:34.918524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:34.919190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:34.919467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:34.919761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:34.920462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:34.920775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:34.922149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:34.936581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 536/1068] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:18:34 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:34.937910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.051907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855d1c3-0699-451a-8bb8-3ea39846e678 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:35.073143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:35.089027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:35.089664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.089944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.090214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.101010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:35.102475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:35.119059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:35.120035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:35.121146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:35.121146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:35.121146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:35.121146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 533/1069] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:34 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:35.127138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.231078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fa1cfa-389e-45b3-96ff-1540885f4798 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:35.263009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:35.263606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.263885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.264158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.264793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:35.265097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:35.266457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:35.278522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 537/1070] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:35.284202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.387397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f97f20fc-84e1-4da3-98d7-2ea7201af662 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:35.421804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:35.422431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.422710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.423010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.430521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:35.431975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:35.443362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:35.444330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:35.445740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 534/1071] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:18:35 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:35.450467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.493495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:35.549123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf39a749-fadf-40c6-a12d-83d6cfdd9002 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:35.582341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:35.583574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.583859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.584127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.584840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:35.585146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:35.586567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e624c1f-3dc5-4c71-b554-fb33b29f4acc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:35.603028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 538/1072] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:18:35 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:35.608755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.643280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:35.714198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee69451d-b387-432e-b9de-f8990d2f6802 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:35.747509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc6be9-ae48-48d4-95cf-f62da067bc7f None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.747789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc6be9-ae48-48d4-95cf-f62da067bc7f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.748059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc6be9-ae48-48d4-95cf-f62da067bc7f None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.749413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 539/1073] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:35.749944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:35.750574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.750887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.751160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.754549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.755472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.755782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.762069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:35.763542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:35.773682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:35.774904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:35.777954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:35.779079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:35.780000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:35.780000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:35.780000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:35.780000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 535/1074] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:35.784656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.843395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 540/1075] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:35 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:35.867625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:35.883514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edadb57f-a84e-4226-8ef8-0e497d5ae610 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:35.916895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:35.917491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:35.917767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:35.918037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:35.918667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:35.919001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:35.920365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:35.931808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 536/1076] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:35.937494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:35.966969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:36.034445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ae4e1e2-9c7f-484a-8d89-0c00ad36fbc1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.066818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.067969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.068237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.068502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.075103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.076524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:36.087296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:36.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:36.089259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 541/1077] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:35 2022] GET /identity/v3/projects/service => generated 88 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:36.094373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.111807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.194441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7faedeba-32f0-4904-b7f5-add64f5ea57c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.198614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:36.207746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc067782-5e63-4ad3-b3e9-f6d2d7f44f1e None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.229019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.229628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.229913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.230183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.230897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:36.231205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.232636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.243283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.243903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.244174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.244437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.246218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 537/1078] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:36 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:36.251861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.256001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:36.256328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:36.260862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:36.261895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:36.262732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 542/1079] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:18:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:36.267796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.353480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-febf254a-95a2-4851-9ea4-e02a700af2e1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.369942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2437b8-c57f-4939-a33b-a4ddfc8f0e54 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.386699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.387383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.387662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.387930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/ab556e3087544c42929aab1b35d145a4/roles/b0033600d8264a8d932bf36ce195ab0c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.390027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=ab556e3087544c42929aab1b35d145a4, role_id=b0033600d8264a8d932bf36ce195ab0c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.391492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.404399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.411974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.411974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.411974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.411974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:36.411974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.411974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.415764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 543/1080] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:18:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:36.421431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.436131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:36.480556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 538/1081] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:18:36 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/ab556e3087544c42929aab1b35d145a4/roles/b0033600d8264a8d932bf36ce195ab0c => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:18:36.523372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-affa28ba-7775-4447-a745-a21244b44c06 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.559660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.560300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.560576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.560847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] PATH_INFO: `/v3/users/a66f2ab311a145349d77271a3e85ec21` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.562021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] RBAC: Authorizing `identity:get_user(user_id=a66f2ab311a145349d77271a3e85ec21)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.563500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.565172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 544/1082] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:36 2022] GET /identity/v3/users/a66f2ab311a145349d77271a3e85ec21 => generated 285 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:36.572042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.640373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:36.683928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3457b408-901c-43b4-a6bd-810a9d2af26a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.724764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ce6dd0b-52ee-4306-8b6b-a76cf5cb610e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.725392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6dd0b-52ee-4306-8b6b-a76cf5cb610e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.725669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6dd0b-52ee-4306-8b6b-a76cf5cb610e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.725939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ce6dd0b-52ee-4306-8b6b-a76cf5cb610e None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.727073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce6dd0b-52ee-4306-8b6b-a76cf5cb610e None admin] RBAC: Authorizing `identity:get_project(project_id=dfce272ab2364d568042aaebcf0102e8)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.728512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ce6dd0b-52ee-4306-8b6b-a76cf5cb610e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.730119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 539/1083] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:18:36 2022] GET /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8 => generated 311 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:36.736308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:36.818915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:36.837608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fab2817-4559-41a4-83a6-59b4885dc8fb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:36.871897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-057149d2-435c-4437-8cdc-372468f8a863 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:36.872543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057149d2-435c-4437-8cdc-372468f8a863 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:36.872818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057149d2-435c-4437-8cdc-372468f8a863 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:36.873085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-057149d2-435c-4437-8cdc-372468f8a863 None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8/users/a66f2ab311a145349d77271a3e85ec21/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:36.874947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057149d2-435c-4437-8cdc-372468f8a863 None admin] RBAC: Authorizing `identity:create_grant(project_id=dfce272ab2364d568042aaebcf0102e8, user_id=a66f2ab311a145349d77271a3e85ec21, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:36.876358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-057149d2-435c-4437-8cdc-372468f8a863 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:36.972015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 545/1084] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:18:36 2022] PUT /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8/users/a66f2ab311a145349d77271a3e85ec21/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:18:38.639853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bd9504-4d12-4568-b283-9548c9c87a71 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:38.640401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bd9504-4d12-4568-b283-9548c9c87a71 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:38.640401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bd9504-4d12-4568-b283-9548c9c87a71 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:38.641732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 540/1085] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:38.647029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd219df5-d001-4662-bba2-682752e989b8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:38.647433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd219df5-d001-4662-bba2-682752e989b8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:38.647587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd219df5-d001-4662-bba2-682752e989b8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:38.666100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fd219df5-d001-4662-bba2-682752e989b8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:38.667328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fd219df5-d001-4662-bba2-682752e989b8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:38.733457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 546/1086] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:38 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:39.010717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bd9504-4d12-4568-b283-9548c9c87a71 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.102903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17189d2-a05a-4be5-a1b0-1a552a9addd2 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.103188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17189d2-a05a-4be5-a1b0-1a552a9addd2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.103465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f17189d2-a05a-4be5-a1b0-1a552a9addd2 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.104803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 547/1087] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:18:39.109907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.110308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.110688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.111769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bd9504-4d12-4568-b283-9548c9c87a71 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.127947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:39.129116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:39.141701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:39.147697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.148328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.148609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.148883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.158763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:39.163890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:39.165075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:39.166141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 541/1088] 10.222.0.64 () {56 vars in 1120 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:39.170864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.193190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:39.197751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 548/1089] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:39 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Sep 27 16:18:39.277486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292fd896-febd-42d3-81d7-185426548b4c None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.310491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.311116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.311397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.311662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.312321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:39.312622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.313996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.322521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 542/1090] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:39.328934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.425218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8228a739-eb96-4de6-8980-94de0c257cd7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.459071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.459696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.459972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.460242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.470196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.472304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.474354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:39.487776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:39.489872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:39.491145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:39.491145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:39.491145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:39.494444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 549/1091] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:39.495296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.534033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:39.553405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:39.578257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd025ebb-3ddc-47dc-b1e5-7cd53c6d3801 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.599730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa3bdd2e-71be-48f5-a65d-de70b1e3afb8 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.611826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.612575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.612853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.613119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.623652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:18:39.628879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:39.633897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:39.634751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 543/1092] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:18:39.635496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.635496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.635496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.635496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.635777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:39.635777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.636820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.636962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.653770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 550/1093] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:39.654796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.740827 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6c5a48-f0a8-4fad-a591-66cf7e602cbc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.754102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:39.754298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2024858c-779d-45be-bc75-d02929942be7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.768374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:39.777282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.777899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.778175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.778444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.779135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:39.779435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.780797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.792795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.792999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 544/1094] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:39.793148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.793148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.793148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.796229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.797991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.799464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:18:39.812096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:39.813335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:39.814691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 551/1095] 10.222.0.64 () {56 vars in 1093 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/users/nova => generated 82 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:39.819744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.901068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432ad0fb-aec4-4c75-a2cc-4797847dbfd9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.918794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1622e654-6f11-4526-a3fd-df9ef49b18e0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:39.933832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.934454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.934728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.935093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] PATH_INFO: `/v3/users/a66f2ab311a145349d77271a3e85ec21` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.936202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] RBAC: Authorizing `identity:get_user(user_id=a66f2ab311a145349d77271a3e85ec21)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.937624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.939202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 545/1096] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/users/a66f2ab311a145349d77271a3e85ec21 => generated 285 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:39.945465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.952155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:39.952755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:39.953036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:39.953303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:39.954017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:39.954319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:39.955755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:39.973004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 552/1097] 10.222.0.64 () {56 vars in 1133 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:39.978512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:39.991622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.007394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.045872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b287764a-077e-4b7d-b1f3-c745bfbc46cd None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.076370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d777c5-97c9-4960-8518-c8c1bf0abd82 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.078291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.078932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.079207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.079473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.111564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.112160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.112426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.112698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.146134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.147627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.179453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.180879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.190329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.209994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.244574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 546/1098] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 299 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 27 16:18:40.250359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:40.277954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 553/1099] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:39 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 300 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Sep 27 16:18:40.283348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:40.350147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550d12ed-fed0-4b3b-87e7-6e834ee4d5b9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.381354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08f59461-5f5f-4754-beb6-e81618e453c4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.382894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.383527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.383803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.384073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.385171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] RBAC: Authorizing `identity:get_project(project_id=dfce272ab2364d568042aaebcf0102e8)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.386585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.388235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 547/1100] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:40.394119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:40.408779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.413241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.418942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.422589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.422589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.422589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.430046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.431554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:18:40.446643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:40.449993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:18:40.450916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:18:40.450916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:18:40.450916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:18:40.450916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 554/1101] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:40.453565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:40.494947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4322087e-09fc-4e52-95f8-afdd5158f5c1 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.528905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.529509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.529786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.530057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.530803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=dfce272ab2364d568042aaebcf0102e8, user.id=a66f2ab311a145349d77271a3e85ec21) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:40.531106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.532532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.538712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 548/1102] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/role_assignments?user.id=a66f2ab311a145349d77271a3e85ec21&scope.project.id=dfce272ab2364d568042aaebcf0102e8&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:40.552970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73461a3f-2121-470d-9920-a5d1170e4622 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.589723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.590338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.590673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.590975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.591627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:40.591937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.593409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.607188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 555/1103] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:40.613004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:40.695927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.720611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f132c68c-f0a0-46e2-9c0d-88a1f328cdfa None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.754664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.756034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.756310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.756580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.764215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.765682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:18:40.777554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:18:40.778561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:18:40.779848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 549/1104] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/projects/service => generated 88 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:18:40.784511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:40.827194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:40.882682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b257895-db44-4e98-8dc5-e7c1bd66ac2b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:40.915725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:40.916325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:40.916600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:40.916869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:40.917546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:40.917846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:40.919334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33702d00-6743-4d95-986b-d2b7be68f67d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:40.931226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 556/1105] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:40.937295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:41.036069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80945e42-629f-4b95-ad22-17b980078272 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:41.069733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:41.070341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:41.070618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:41.070917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:41.071633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] RBAC: Adding query filter params (role.id=b0033600d8264a8d932bf36ce195ab0c, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=ab556e3087544c42929aab1b35d145a4) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:41.071940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:41.073389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e880a38-140f-4b24-9533-71b11de72d7c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:41.076394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:41.079731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 550/1106] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:40 2022] GET /identity/v3/role_assignments?user.id=ab556e3087544c42929aab1b35d145a4&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=b0033600d8264a8d932bf36ce195ab0c => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:42.723026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b72fc-d050-4e6e-a4dc-0c1ec91aadae None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:42.723670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b72fc-d050-4e6e-a4dc-0c1ec91aadae None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:42.723670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-373b72fc-d050-4e6e-a4dc-0c1ec91aadae None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:42.724886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 557/1107] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:42.730166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:42.730587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:42.730697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:42.749054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:42.750242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:42.814300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 551/1108] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:42 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:43.080628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373b72fc-d050-4e6e-a4dc-0c1ec91aadae None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:43.178721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-373b72fc-d050-4e6e-a4dc-0c1ec91aadae None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:43.180218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:43.211087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41f17be-b040-435b-a6cb-8b339068f398 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:43.211717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41f17be-b040-435b-a6cb-8b339068f398 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:43.211991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41f17be-b040-435b-a6cb-8b339068f398 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:43.212259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41f17be-b040-435b-a6cb-8b339068f398 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:43.213343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41f17be-b040-435b-a6cb-8b339068f398 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:43.214750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41f17be-b040-435b-a6cb-8b339068f398 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:43.216368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 558/1109] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:43.222135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:43.318113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3843a59b-8102-4ad7-8504-2c9dedb5d7a9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:43.346919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:43.353309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2ecc6d-9050-430b-80a5-c357c96d73ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:43.353918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2ecc6d-9050-430b-80a5-c357c96d73ac None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:43.354197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2ecc6d-9050-430b-80a5-c357c96d73ac None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:43.354469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2ecc6d-9050-430b-80a5-c357c96d73ac None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:43.355154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2ecc6d-9050-430b-80a5-c357c96d73ac None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:43.356576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2ecc6d-9050-430b-80a5-c357c96d73ac None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:43.439470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 552/1110] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:18:43 2022] POST /identity/v3/users => generated 286 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:18:45.639702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0728a8-c85a-4eca-a51c-fe66700af959 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:45.640490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0728a8-c85a-4eca-a51c-fe66700af959 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:45.640667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0728a8-c85a-4eca-a51c-fe66700af959 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:45.642034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 559/1111] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:45 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:45.647666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:45.648096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:45.648201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:45.666177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:45.667458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:45.734162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 553/1112] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:45 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:46.012779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0728a8-c85a-4eca-a51c-fe66700af959 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:46.109539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0728a8-c85a-4eca-a51c-fe66700af959 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:46.141435 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:46.142037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:46.142309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:46.142574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] PATH_INFO: `/v3/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:46.143943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] RBAC: Authorizing `identity:get_role(role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:46.145963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:46.147268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 560/1113] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:46 2022] GET /identity/v3/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 223 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:46.153065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:46.250052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06498fb-8b31-48b7-8959-77bb349dc10c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:46.270755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:46.287707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:46.288326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:46.288603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:46.293528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] PATH_INFO: `/v3/users/c96f3557b3e842a08790b426a0cb5302` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:46.302352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] RBAC: Authorizing `identity:get_user(user_id=c96f3557b3e842a08790b426a0cb5302)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:46.303885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:46.305465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 554/1114] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:46 2022] GET /identity/v3/users/c96f3557b3e842a08790b426a0cb5302 => generated 286 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:46.311398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:46.350846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:46.407958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9077642a-5634-4053-8001-bab8155dd81f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:46.440516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:46.441104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:46.442011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:46.442288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:46.506392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:46.507855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:46.543843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:46.604832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 561/1115] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:46 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:46.611444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:46.706425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe93518c-aa9b-49bb-ac90-cdf22e8b941a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:46.739026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12430175-3be8-4210-9f3d-c3fc18f346f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:46.739638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12430175-3be8-4210-9f3d-c3fc18f346f5 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:46.739911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12430175-3be8-4210-9f3d-c3fc18f346f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:46.740176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12430175-3be8-4210-9f3d-c3fc18f346f5 None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:46.741261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12430175-3be8-4210-9f3d-c3fc18f346f5 None admin] RBAC: Authorizing `identity:get_project(project_id=dfce272ab2364d568042aaebcf0102e8)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:46.742668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12430175-3be8-4210-9f3d-c3fc18f346f5 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:46.744282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 555/1116] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:18:46 2022] GET /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:46.750310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:46.850186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac87bb39-e1d6-41c0-a0ce-23f780a37917 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:46.882321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:46.882938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:46.883213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:46.883479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:46.884187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] RBAC: Adding query filter params (role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=dfce272ab2364d568042aaebcf0102e8, user.id=c96f3557b3e842a08790b426a0cb5302) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:46.884492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:46.885900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f94500b5-e452-471a-aeba-9d65149a128b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:46.891211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 562/1117] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:46 2022] GET /identity/v3/role_assignments?user.id=c96f3557b3e842a08790b426a0cb5302&scope.project.id=dfce272ab2364d568042aaebcf0102e8&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:49.000968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63db686-1609-42fa-ad97-48d08deef69e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:49.001494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63db686-1609-42fa-ad97-48d08deef69e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:49.001494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d63db686-1609-42fa-ad97-48d08deef69e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:49.002857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 556/1118] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:49.008666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:49.009066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:49.009202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:49.015396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:49.028292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:49.029481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:49.093338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 563/1119] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:49 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:49.361094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63db686-1609-42fa-ad97-48d08deef69e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:49.405419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:49.461660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d63db686-1609-42fa-ad97-48d08deef69e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:49.494567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:49.495211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:49.495488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:49.495756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] PATH_INFO: `/v3/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:49.497110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] RBAC: Authorizing `identity:get_role(role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:49.498448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:49.499774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 557/1120] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:49 2022] GET /identity/v3/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 223 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:49.506232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:49.601828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3a409b-2eef-4f33-8061-2a01a4d63c73 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:49.630056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:49.634688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:49.635327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:49.635600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:49.635864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] PATH_INFO: `/v3/users/c96f3557b3e842a08790b426a0cb5302` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:49.636945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] RBAC: Authorizing `identity:get_user(user_id=c96f3557b3e842a08790b426a0cb5302)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:49.638346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:49.639918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 564/1121] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:49 2022] GET /identity/v3/users/c96f3557b3e842a08790b426a0cb5302 => generated 286 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:49.645676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:49.743616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1359476-174a-491a-9850-7ffc5903578a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:49.753456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:49.775633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a999cb05-a3a4-444b-a053-333924fa7016 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:49.776242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999cb05-a3a4-444b-a053-333924fa7016 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:49.776518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999cb05-a3a4-444b-a053-333924fa7016 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:49.776787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a999cb05-a3a4-444b-a053-333924fa7016 None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:49.777865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999cb05-a3a4-444b-a053-333924fa7016 None admin] RBAC: Authorizing `identity:get_project(project_id=dfce272ab2364d568042aaebcf0102e8)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:49.779315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a999cb05-a3a4-444b-a053-333924fa7016 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:49.781097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 558/1122] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:18:49 2022] GET /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:18:49.787301 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:49.883064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0589b94c-75cd-4d18-92e5-28f4fe948d94 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:49.915451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f11282-187e-4162-b085-ccdb12c7892a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:49.916088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f11282-187e-4162-b085-ccdb12c7892a None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:49.916365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f11282-187e-4162-b085-ccdb12c7892a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:49.916636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f11282-187e-4162-b085-ccdb12c7892a None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8/users/c96f3557b3e842a08790b426a0cb5302/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:49.918728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f11282-187e-4162-b085-ccdb12c7892a None admin] RBAC: Authorizing `identity:create_grant(project_id=dfce272ab2364d568042aaebcf0102e8, user_id=c96f3557b3e842a08790b426a0cb5302, role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:49.920185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80f11282-187e-4162-b085-ccdb12c7892a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:50.016351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 565/1123] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:18:49 2022] PUT /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8/users/c96f3557b3e842a08790b426a0cb5302/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 0 bytes in 229 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:18:52.132851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e968e17d-660d-41ac-9bab-b217b49a5129 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:52.133406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e968e17d-660d-41ac-9bab-b217b49a5129 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:52.133406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e968e17d-660d-41ac-9bab-b217b49a5129 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:52.134723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 559/1124] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:52.139873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:52.140244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:52.140418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:52.159248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:52.160454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:52.213900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:52.224891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 566/1125] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:52.493506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e968e17d-660d-41ac-9bab-b217b49a5129 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:52.591869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e968e17d-660d-41ac-9bab-b217b49a5129 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:52.625164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:52.625771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:52.626043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:52.626306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] PATH_INFO: `/v3/roles/8d847abe3bdc4ad18b5c4d1f251d1e09` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:52.627675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] RBAC: Authorizing `identity:get_role(role_id=8d847abe3bdc4ad18b5c4d1f251d1e09)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:52.629028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:52.630336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 560/1126] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:52 2022] GET /identity/v3/roles/8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 223 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:52.636051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:52.730859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78671cee-c9d4-4a33-a4bf-2bb2f093bd7b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:52.763386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:52.763941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:52.764216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:52.764483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] PATH_INFO: `/v3/users/c96f3557b3e842a08790b426a0cb5302` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:52.765581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] RBAC: Authorizing `identity:get_user(user_id=c96f3557b3e842a08790b426a0cb5302)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:52.767017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:52.768555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 567/1127] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:18:52 2022] GET /identity/v3/users/c96f3557b3e842a08790b426a0cb5302 => generated 286 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:52.774324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:52.819775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:52.871472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7141833-fd41-4b24-a1ec-6fe46331d6ae None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:52.903310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:52.903912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:52.904190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:52.904460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:52.969709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:52.971168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:53.014056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:53.065721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 561/1128] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:18:52 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:18:53.072184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:53.127136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:53.165892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd61e77-9dda-4aa1-ac7c-5cd567f11389 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:53.199546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c5f297d-bd60-403a-bf88-09c11461eaba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:53.200157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f297d-bd60-403a-bf88-09c11461eaba None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:53.200433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f297d-bd60-403a-bf88-09c11461eaba None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:53.200705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c5f297d-bd60-403a-bf88-09c11461eaba None admin] PATH_INFO: `/v3/projects/dfce272ab2364d568042aaebcf0102e8` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:53.201796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5f297d-bd60-403a-bf88-09c11461eaba None admin] RBAC: Authorizing `identity:get_project(project_id=dfce272ab2364d568042aaebcf0102e8)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:53.203250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c5f297d-bd60-403a-bf88-09c11461eaba None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:53.204837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 568/1129] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:18:53 2022] GET /identity/v3/projects/dfce272ab2364d568042aaebcf0102e8 => generated 311 bytes in 133 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:53.210916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:53.307824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-651d1d4d-304f-453f-82bd-42eacceb341f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:53.341110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:53.341711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:53.341987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:53.342256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:53.342993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] RBAC: Adding query filter params (role.id=8d847abe3bdc4ad18b5c4d1f251d1e09, scope.project.id=dfce272ab2364d568042aaebcf0102e8, user.id=c96f3557b3e842a08790b426a0cb5302) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:18:53.343298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:53.344726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de922d90-5de7-469f-91f1-0f1d19b3b47d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:53.351604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 562/1130] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:18:53 2022] GET /identity/v3/role_assignments?user.id=c96f3557b3e842a08790b426a0cb5302&scope.project.id=dfce272ab2364d568042aaebcf0102e8&role.id=8d847abe3bdc4ad18b5c4d1f251d1e09 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:55.524693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f16699-40b1-4eef-a098-5fa531e89cd7 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:55.525245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f16699-40b1-4eef-a098-5fa531e89cd7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:55.525245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04f16699-40b1-4eef-a098-5fa531e89cd7 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:55.526558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 569/1131] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:55.531779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:55.532198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:55.532349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:55.549989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:55.551740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:55.554435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:55.616973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 563/1132] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:55 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:55.882983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f16699-40b1-4eef-a098-5fa531e89cd7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:55.981198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04f16699-40b1-4eef-a098-5fa531e89cd7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:56.005918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:56.013809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50048358-e9fe-401c-b9a6-d971b8cea895 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:56.015140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50048358-e9fe-401c-b9a6-d971b8cea895 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:56.015140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50048358-e9fe-401c-b9a6-d971b8cea895 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:56.015140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50048358-e9fe-401c-b9a6-d971b8cea895 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:56.016084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50048358-e9fe-401c-b9a6-d971b8cea895 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:56.017493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50048358-e9fe-401c-b9a6-d971b8cea895 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:56.019094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 570/1133] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:55 2022] GET /identity/v3/domains/default => generated 199 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:56.025053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:56.119857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbaebd20-7734-4a9a-af76-6a4483c9efa1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:56.152035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bee6aad-ee61-4f47-9e78-564c99510c88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:56.152567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee6aad-ee61-4f47-9e78-564c99510c88 None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:56.152840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee6aad-ee61-4f47-9e78-564c99510c88 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:56.153105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bee6aad-ee61-4f47-9e78-564c99510c88 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:56.153739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee6aad-ee61-4f47-9e78-564c99510c88 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:56.155172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bee6aad-ee61-4f47-9e78-564c99510c88 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:56.165715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:56.228033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 564/1134] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:18:56 2022] POST /identity/v3/projects => generated 311 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:18:58.397704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3832783-88ad-4b68-a735-2eb474983b80 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:58.398216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3832783-88ad-4b68-a735-2eb474983b80 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:58.398308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3832783-88ad-4b68-a735-2eb474983b80 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:58.399718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 571/1135] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:18:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:18:58.404829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:58.405262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:58.405356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:58.423715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:18:58.424911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:18:58.488929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 565/1136] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:18:58 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:18:58.755999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3832783-88ad-4b68-a735-2eb474983b80 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:58.852191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3832783-88ad-4b68-a735-2eb474983b80 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:58.884794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2305efba-4601-48d6-ba3d-83f27421e918 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:58.885398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305efba-4601-48d6-ba3d-83f27421e918 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:58.885674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305efba-4601-48d6-ba3d-83f27421e918 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:58.885941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2305efba-4601-48d6-ba3d-83f27421e918 None admin] PATH_INFO: `/v3/domains/default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:58.887046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2305efba-4601-48d6-ba3d-83f27421e918 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:58.888453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2305efba-4601-48d6-ba3d-83f27421e918 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:58.889993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 572/1137] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:18:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:18:58.896155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:18:58.994417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-751abf34-f9f3-4314-a222-c80100664a7d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:18:59.016387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:18:59.028556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5698c74b-de34-4cd8-981d-b4afe3a97e9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:18:59.029157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5698c74b-de34-4cd8-981d-b4afe3a97e9e None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:18:59.029440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5698c74b-de34-4cd8-981d-b4afe3a97e9e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:18:59.043181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5698c74b-de34-4cd8-981d-b4afe3a97e9e None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:18:59.043181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5698c74b-de34-4cd8-981d-b4afe3a97e9e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:18:59.043181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5698c74b-de34-4cd8-981d-b4afe3a97e9e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:18:59.169986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 566/1138] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:18:58 2022] POST /identity/v3/users => generated 286 bytes in 274 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:19:01.373260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8089743-239b-4518-a156-acb59a5375b2 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:01.373720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8089743-239b-4518-a156-acb59a5375b2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:01.373834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8089743-239b-4518-a156-acb59a5375b2 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:01.375152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 573/1139] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:01.380900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:01.381333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:01.381436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:01.399264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:01.400426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:01.465815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 567/1140] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:01.741226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8089743-239b-4518-a156-acb59a5375b2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:01.801648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:01.839314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8089743-239b-4518-a156-acb59a5375b2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:01.871075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:01.871694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:01.871972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:01.872240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:01.882877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:19:01.887645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:01.888750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:19:01.889778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 574/1141] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:19:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:01.895366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:01.996098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eabc25a-226d-4377-89ed-1cdd6fad50e3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:02.005534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:02.031217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:02.031821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:02.032094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:02.032362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:02.033013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:02.033311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:02.034669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:02.043654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 568/1142] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:19:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:02.049681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:02.151566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b70791ae-5d50-48cd-9006-e4fa980e12e1 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:02.173416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:02.185698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:02.186326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:02.186607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:02.186910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] PATH_INFO: `/v3/users/f1157c13767d4a9ea1afeba0f149ba78` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:02.203726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] RBAC: Authorizing `identity:get_user(user_id=f1157c13767d4a9ea1afeba0f149ba78)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:02.205541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:02.208319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 575/1143] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:19:02 2022] GET /identity/v3/users/f1157c13767d4a9ea1afeba0f149ba78 => generated 286 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:02.215066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:02.333315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a88c3bd-ef42-4d21-bc6a-8b2b441215b3 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:02.375603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:02.376448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:02.376730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:02.377000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:02.417363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:02.448513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:02.450492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:02.555159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 569/1144] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:19:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:19:02.561224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:02.670251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c7b78-0b6f-4e53-869d-d66c144e92bc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:02.704305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65bbafc1-f111-4eb3-9698-ba1b3f8d62ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:02.704923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bbafc1-f111-4eb3-9698-ba1b3f8d62ac None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:02.705197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bbafc1-f111-4eb3-9698-ba1b3f8d62ac None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:02.705466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65bbafc1-f111-4eb3-9698-ba1b3f8d62ac None admin] PATH_INFO: `/v3/projects/e2c96348df25460188f9289e03724ff0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:02.707374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bbafc1-f111-4eb3-9698-ba1b3f8d62ac None admin] RBAC: Authorizing `identity:get_project(project_id=e2c96348df25460188f9289e03724ff0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:02.709421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65bbafc1-f111-4eb3-9698-ba1b3f8d62ac None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:02.711148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 576/1145] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:19:02 2022] GET /identity/v3/projects/e2c96348df25460188f9289e03724ff0 => generated 311 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:02.717470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:02.787108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:02.819001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b92d2872-01a3-4662-9341-fbb3f4cad8fc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:02.851498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:02.852092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:02.852367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:02.852637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:02.853461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=e2c96348df25460188f9289e03724ff0, user.id=f1157c13767d4a9ea1afeba0f149ba78) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:02.853771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:02.855214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5faf37-7894-4e4b-8520-b6187023ebcc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:02.860455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 570/1146] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:19:02 2022] GET /identity/v3/role_assignments?user.id=f1157c13767d4a9ea1afeba0f149ba78&scope.project.id=e2c96348df25460188f9289e03724ff0&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:04.983952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7d1dc-6812-4754-b87d-fcce055c6b71 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:04.984520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7d1dc-6812-4754-b87d-fcce055c6b71 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:04.984520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fe7d1dc-6812-4754-b87d-fcce055c6b71 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:04.985831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 577/1147] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:04.991448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:04.991889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:04.992020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:05.010447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:05.011657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:05.076332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 571/1148] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:04 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:05.342927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7d1dc-6812-4754-b87d-fcce055c6b71 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:05.391405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:05.442866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fe7d1dc-6812-4754-b87d-fcce055c6b71 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:05.474937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:05.475536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:05.475808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:05.476069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:05.485613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:19:05.490929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:05.492035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:19:05.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 578/1149] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:19:05 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:05.497981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:05.558917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:05.595070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576444d9-f1c6-48dd-9075-27ed959b4315 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:05.628050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:05.628640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:05.628916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:05.629185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:05.629835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:05.630136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:05.631523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:05.639673 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 572/1150] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:19:05 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:05.645544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:05.743181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0fcfc4-1179-4b14-8aa7-efc454bca336 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:05.751960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:05.776373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:05.776990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:05.777268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:05.777538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] PATH_INFO: `/v3/users/f1157c13767d4a9ea1afeba0f149ba78` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:05.778694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] RBAC: Authorizing `identity:get_user(user_id=f1157c13767d4a9ea1afeba0f149ba78)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:05.780141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:05.781708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 579/1151] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:19:05 2022] GET /identity/v3/users/f1157c13767d4a9ea1afeba0f149ba78 => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:05.788078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:05.884011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b796bb99-ac8c-47c8-afb9-e57fa5f8967f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:05.909700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:05.918086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9efffc-e7a3-493d-ba12-918d1ea251a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:05.925373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9efffc-e7a3-493d-ba12-918d1ea251a0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:05.925373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9efffc-e7a3-493d-ba12-918d1ea251a0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:05.925373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9efffc-e7a3-493d-ba12-918d1ea251a0 None admin] PATH_INFO: `/v3/projects/e2c96348df25460188f9289e03724ff0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:05.925373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9efffc-e7a3-493d-ba12-918d1ea251a0 None admin] RBAC: Authorizing `identity:get_project(project_id=e2c96348df25460188f9289e03724ff0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:05.925373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9efffc-e7a3-493d-ba12-918d1ea251a0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:05.925373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 573/1152] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:19:05 2022] GET /identity/v3/projects/e2c96348df25460188f9289e03724ff0 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:05.929191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:06.026688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc22b2aa-4316-45f8-89a3-3cab5861350d None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:06.059571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfec388-c087-4853-a1d8-984cac11d131 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:06.060209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfec388-c087-4853-a1d8-984cac11d131 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:06.060489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfec388-c087-4853-a1d8-984cac11d131 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:06.060767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfec388-c087-4853-a1d8-984cac11d131 None admin] PATH_INFO: `/v3/projects/e2c96348df25460188f9289e03724ff0/users/f1157c13767d4a9ea1afeba0f149ba78/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:06.062631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfec388-c087-4853-a1d8-984cac11d131 None admin] RBAC: Authorizing `identity:create_grant(project_id=e2c96348df25460188f9289e03724ff0, user_id=f1157c13767d4a9ea1afeba0f149ba78, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:06.064073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cfec388-c087-4853-a1d8-984cac11d131 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:06.152843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 580/1153] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:19:05 2022] PUT /identity/v3/projects/e2c96348df25460188f9289e03724ff0/users/f1157c13767d4a9ea1afeba0f149ba78/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:19:08.270563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9fea2-860f-4e9d-be4f-522741d36e84 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:08.271229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9fea2-860f-4e9d-be4f-522741d36e84 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:08.271229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90a9fea2-860f-4e9d-be4f-522741d36e84 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:08.272461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 574/1154] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:08.277623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:08.278077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:08.278177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:08.296945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:08.298114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:08.325964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:08.364111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 581/1155] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:08 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:08.633869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9fea2-860f-4e9d-be4f-522741d36e84 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:08.735013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90a9fea2-860f-4e9d-be4f-522741d36e84 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:08.767904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:08.769499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:08.769781 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:08.770057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:08.779891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:19:08.785844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:08.787045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:19:08.787968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 575/1156] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:19:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 154 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:08.792980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:08.887026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d08ff06-fc04-4628-ace6-ef667fc5aad7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:08.919977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:08.920562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:08.920834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:08.921099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:08.921743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:08.922038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:08.923411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:08.931119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 582/1157] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:19:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:08.936858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:08.990952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:09.035733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbc4dec2-f76f-41f5-a13b-3a5c8170f6c7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:09.069004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:09.069622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:09.069898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:09.070168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] PATH_INFO: `/v3/users/f1157c13767d4a9ea1afeba0f149ba78` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:09.071285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] RBAC: Authorizing `identity:get_user(user_id=f1157c13767d4a9ea1afeba0f149ba78)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:09.072713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:09.074527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 576/1158] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:19:08 2022] GET /identity/v3/users/f1157c13767d4a9ea1afeba0f149ba78 => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:09.080262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:09.108072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:09.176620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e613ac6-d494-4dc3-9d62-9526d5be3be2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:09.208065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:09.208648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:09.208919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:09.209183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:09.271889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:09.273302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:09.298395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:09.366810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 583/1159] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:19:09 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 287 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:19:09.372669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:09.470630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d82795bd-faa5-484e-ad21-b95d6ff7673e None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:09.494230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:09.505847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0280e05-2fbc-4d00-9d68-7a4ca99a5fd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:09.507316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0280e05-2fbc-4d00-9d68-7a4ca99a5fd4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:09.507316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0280e05-2fbc-4d00-9d68-7a4ca99a5fd4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:09.507316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0280e05-2fbc-4d00-9d68-7a4ca99a5fd4 None admin] PATH_INFO: `/v3/projects/e2c96348df25460188f9289e03724ff0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:09.508195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0280e05-2fbc-4d00-9d68-7a4ca99a5fd4 None admin] RBAC: Authorizing `identity:get_project(project_id=e2c96348df25460188f9289e03724ff0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:09.509645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0280e05-2fbc-4d00-9d68-7a4ca99a5fd4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:09.511361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 577/1160] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:19:09 2022] GET /identity/v3/projects/e2c96348df25460188f9289e03724ff0 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:09.517504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:09.611811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7074ff2-8b8e-4bac-92ba-2fe61cf3796f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:09.637675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:09.645055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:09.652960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:09.652960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:09.652960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:09.652960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=e2c96348df25460188f9289e03724ff0, user.id=f1157c13767d4a9ea1afeba0f149ba78) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:09.652960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:09.652960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9944055b-b407-4abd-9187-b70c1b24e3ac None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:09.654108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 584/1161] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:19:09 2022] GET /identity/v3/role_assignments?user.id=f1157c13767d4a9ea1afeba0f149ba78&scope.project.id=e2c96348df25460188f9289e03724ff0&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:11.799067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92e5a29-3601-402d-954c-11bc97c69029 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:11.800179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92e5a29-3601-402d-954c-11bc97c69029 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:11.800314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92e5a29-3601-402d-954c-11bc97c69029 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:11.801654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 578/1162] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:11.806829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:11.807247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:11.807370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:11.825900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:11.827100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:11.891015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 585/1163] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:11 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:12.155186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92e5a29-3601-402d-954c-11bc97c69029 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:12.256882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92e5a29-3601-402d-954c-11bc97c69029 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:12.289030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:12.289623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:12.289896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:12.290161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:12.344995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:12.355517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:12.359068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:12.451881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 579/1164] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:19:12 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 297 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:19:12.458319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:12.554519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6001f4-067e-45f7-b718-1e40739d0e68 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:12.576115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:12.587847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:12.588469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:12.588745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:12.589013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:12.595497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:12.597636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-f5d0973f-2fba-430b-bba2-04ced38746f5 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:12.609007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:12.610338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:19:12.611528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 586/1165] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:19:12 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:12.616203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:12.705050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:12.715223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cc135a9-1fb4-4620-90d5-8fe4f432f3a6 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:12.748908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:12.749501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:12.749779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:12.750051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:12.750686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:12.751047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:12.753007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05fade75-56f7-4cb9-9b7d-c54e1c48d950 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:12.760268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 580/1166] 10.222.0.64 () {56 vars in 1115 bytes} [Tue Sep 27 16:19:12 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:14.879615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c250a8-0851-4ede-8d65-b6d2d54b3819 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:14.880152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c250a8-0851-4ede-8d65-b6d2d54b3819 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:14.880152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9c250a8-0851-4ede-8d65-b6d2d54b3819 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:14.881479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 587/1167] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:14.886839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337145e4-3ddc-497b-848f-0fe7e1b6eb15 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:14.887268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337145e4-3ddc-497b-848f-0fe7e1b6eb15 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:14.887370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-337145e4-3ddc-497b-848f-0fe7e1b6eb15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:14.905508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-337145e4-3ddc-497b-848f-0fe7e1b6eb15 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:14.906677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-337145e4-3ddc-497b-848f-0fe7e1b6eb15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:14.969948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 581/1168] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:14 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:15.243184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c250a8-0851-4ede-8d65-b6d2d54b3819 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:15.339608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9c250a8-0851-4ede-8d65-b6d2d54b3819 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:15.372476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9268ccbd-d273-4d72-8568-9ebd45b095b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:15.373068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268ccbd-d273-4d72-8568-9ebd45b095b0 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:15.373345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268ccbd-d273-4d72-8568-9ebd45b095b0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:15.373615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9268ccbd-d273-4d72-8568-9ebd45b095b0 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:15.374267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268ccbd-d273-4d72-8568-9ebd45b095b0 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:15.375641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9268ccbd-d273-4d72-8568-9ebd45b095b0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:15.450598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 588/1169] 10.222.0.64 () {60 vars in 1138 bytes} [Tue Sep 27 16:19:15 2022] POST /identity/v3/domains => generated 267 bytes in 208 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:19:17.680542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4047209e-bb77-4ec6-bc49-c7035f53a1cb None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:17.681065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4047209e-bb77-4ec6-bc49-c7035f53a1cb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:17.681065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4047209e-bb77-4ec6-bc49-c7035f53a1cb None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:17.682414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 582/1170] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:17.687890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:17.688288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:17.688437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:17.707597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:17.708783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:17.742581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:17.773983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 589/1171] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:18.041756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4047209e-bb77-4ec6-bc49-c7035f53a1cb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:18.126318 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:18.142594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4047209e-bb77-4ec6-bc49-c7035f53a1cb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:18.178289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c075a8f-ea4c-471e-8fb7-1394b072787e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:18.178944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c075a8f-ea4c-471e-8fb7-1394b072787e None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:18.179226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c075a8f-ea4c-471e-8fb7-1394b072787e None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:18.179499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c075a8f-ea4c-471e-8fb7-1394b072787e None admin] PATH_INFO: `/v3/domains/6ba786b3ca8046eda431b6ed9ab8e906` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:18.190116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c075a8f-ea4c-471e-8fb7-1394b072787e None admin] RBAC: Authorizing `identity:get_domain(domain_id=6ba786b3ca8046eda431b6ed9ab8e906)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:18.191613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c075a8f-ea4c-471e-8fb7-1394b072787e None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:18.193222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 583/1172] 10.222.0.64 () {56 vars in 1183 bytes} [Tue Sep 27 16:19:18 2022] GET /identity/v3/domains/6ba786b3ca8046eda431b6ed9ab8e906 => generated 267 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:18.199327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:18.294400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ab32d8-405d-4081-be48-15a920b34c64 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:18.327347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41c5557a-b58e-4dca-8c10-6b3d1d439eeb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:18.327944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c5557a-b58e-4dca-8c10-6b3d1d439eeb None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:18.328219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c5557a-b58e-4dca-8c10-6b3d1d439eeb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:18.328488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41c5557a-b58e-4dca-8c10-6b3d1d439eeb None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:18.329129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c5557a-b58e-4dca-8c10-6b3d1d439eeb None admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:18.330611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41c5557a-b58e-4dca-8c10-6b3d1d439eeb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:18.428061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 590/1173] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:19:18 2022] POST /identity/v3/projects => generated 361 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:19:20.586548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf70b702-de78-45bd-8666-8eff17572343 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:20.587103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf70b702-de78-45bd-8666-8eff17572343 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:20.587222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf70b702-de78-45bd-8666-8eff17572343 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:20.588454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 584/1174] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:20.593851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:20.594236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:20.594334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:20.613512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:20.614683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:20.631040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:20.679130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 591/1175] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:20 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:20.950589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf70b702-de78-45bd-8666-8eff17572343 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:21.052290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf70b702-de78-45bd-8666-8eff17572343 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:21.084899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daba6f3a-6e33-436d-bc7c-bea504a0a887 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:21.085513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daba6f3a-6e33-436d-bc7c-bea504a0a887 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:21.085789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daba6f3a-6e33-436d-bc7c-bea504a0a887 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:21.086061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daba6f3a-6e33-436d-bc7c-bea504a0a887 None admin] PATH_INFO: `/v3/domains/6ba786b3ca8046eda431b6ed9ab8e906` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:21.087169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daba6f3a-6e33-436d-bc7c-bea504a0a887 None admin] RBAC: Authorizing `identity:get_domain(domain_id=6ba786b3ca8046eda431b6ed9ab8e906)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:21.088591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daba6f3a-6e33-436d-bc7c-bea504a0a887 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:21.090134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 585/1176] 10.222.0.64 () {56 vars in 1183 bytes} [Tue Sep 27 16:19:20 2022] GET /identity/v3/domains/6ba786b3ca8046eda431b6ed9ab8e906 => generated 267 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:21.096501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:21.191357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcef2a6-309b-4304-bcb1-d55c3ef523bd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:21.223658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e9e3b34-f98b-4cd5-ba81-a6ae962d7fe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:21.224188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e3b34-f98b-4cd5-ba81-a6ae962d7fe7 None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:21.224461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e3b34-f98b-4cd5-ba81-a6ae962d7fe7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:21.224726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e9e3b34-f98b-4cd5-ba81-a6ae962d7fe7 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:21.225361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9e3b34-f98b-4cd5-ba81-a6ae962d7fe7 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:21.226803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e9e3b34-f98b-4cd5-ba81-a6ae962d7fe7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:21.311609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 592/1177] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:19:21 2022] POST /identity/v3/users => generated 311 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:19:23.494577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18519adc-48fb-4117-862d-8f2d6d3c7b2f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:23.495174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18519adc-48fb-4117-862d-8f2d6d3c7b2f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:23.495174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18519adc-48fb-4117-862d-8f2d6d3c7b2f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:23.496456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 586/1178] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:23.502284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:23.502884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:23.502988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:23.521297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:23.522478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:23.525177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:23.586769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 593/1179] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:23 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:23.856965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18519adc-48fb-4117-862d-8f2d6d3c7b2f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:23.871980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:23.956656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18519adc-48fb-4117-862d-8f2d6d3c7b2f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:23.989806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:23.990417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:23.990692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:23.990985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:24.000704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:19:24.006111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:24.007385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:19:24.008533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 587/1180] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:19:23 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:24.013234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:24.108374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41d528bd-d0a5-460d-a17d-c11c5d14bb72 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:24.141817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:24.142489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:24.142761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:24.143062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:24.143712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:24.144012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:24.145363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:24.153066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 594/1181] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:19:24 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:24.159573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:24.227490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:24.256004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-178ce130-4885-47f1-a529-e6f540450615 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:24.289754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:24.290375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:24.290659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:24.290969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] PATH_INFO: `/v3/users/db742fe9329d4169969174910605dc94` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:24.304293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] RBAC: Authorizing `identity:get_user(user_id=db742fe9329d4169969174910605dc94)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:24.305732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:24.307344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 588/1182] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:19:24 2022] GET /identity/v3/users/db742fe9329d4169969174910605dc94 => generated 311 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:24.313059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:24.316074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:24.411532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f733e99a-731b-4ea8-8b43-be3e227748eb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:24.443469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:24.444067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:24.444347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:24.444618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:24.508924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:24.510362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:24.512125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:24.605751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 595/1183] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:19:24 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:19:24.611882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:24.708641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af64a374-0f4b-4c26-b641-2c120b1a0a75 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:24.736703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:24.742302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc1aeb2-796d-4157-89ab-70f73c7ded44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:24.743094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc1aeb2-796d-4157-89ab-70f73c7ded44 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:24.743378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc1aeb2-796d-4157-89ab-70f73c7ded44 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:24.743651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc1aeb2-796d-4157-89ab-70f73c7ded44 None admin] PATH_INFO: `/v3/projects/28e7a09459514ff78540606599ff1d26` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:24.744746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc1aeb2-796d-4157-89ab-70f73c7ded44 None admin] RBAC: Authorizing `identity:get_project(project_id=28e7a09459514ff78540606599ff1d26)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:24.746178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc1aeb2-796d-4157-89ab-70f73c7ded44 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:24.747826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 589/1184] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:19:24 2022] GET /identity/v3/projects/28e7a09459514ff78540606599ff1d26 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:24.754154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:24.851301 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf211921-55d1-4963-b14e-b22e3f2f09de None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:24.852827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:24.885960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:24.886552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:24.886854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:24.887127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:24.887836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=28e7a09459514ff78540606599ff1d26, user.id=db742fe9329d4169969174910605dc94) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:24.888140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:24.889558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-223a89ac-4301-49aa-a343-99102e15617a None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:24.896977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 596/1185] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:19:24 2022] GET /identity/v3/role_assignments?user.id=db742fe9329d4169969174910605dc94&scope.project.id=28e7a09459514ff78540606599ff1d26&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:27.005610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550aa110-323e-4b11-9798-a3f6462e3474 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:27.006180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550aa110-323e-4b11-9798-a3f6462e3474 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:27.006180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-550aa110-323e-4b11-9798-a3f6462e3474 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:27.007626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 590/1186] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:27.013058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:27.013476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:27.013575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:27.031205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:27.032384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:27.096054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 597/1187] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:27 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:27.367251 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550aa110-323e-4b11-9798-a3f6462e3474 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:27.466140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-550aa110-323e-4b11-9798-a3f6462e3474 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:27.499261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:27.499870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:27.500143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:27.500409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:27.511067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:19:27.515762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:27.516836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:19:27.517911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 591/1188] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:19:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:27.523027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:27.610738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:27.616870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff61e10d-c5cf-4a17-a7e6-faf2f3e0aafa None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:27.649250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:27.649843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:27.650120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:27.650387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:27.651068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:27.651371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:27.652719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:27.660630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 598/1189] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:19:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:27.666556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:27.723743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:27.764692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2fae62-ab7b-429f-8dbd-736443cc6939 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:27.796666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:27.797279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:27.797557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:27.797828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] PATH_INFO: `/v3/users/db742fe9329d4169969174910605dc94` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:27.799550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] RBAC: Authorizing `identity:get_user(user_id=db742fe9329d4169969174910605dc94)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:27.800979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:27.802531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 592/1190] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:19:27 2022] GET /identity/v3/users/db742fe9329d4169969174910605dc94 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:27.808224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:27.906871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82c3da02-2c4c-495f-89fb-3b294fe623c2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:27.939311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1a8d13-6a55-456a-9dea-30af1147c50d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:27.939916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a8d13-6a55-456a-9dea-30af1147c50d None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:27.940188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a8d13-6a55-456a-9dea-30af1147c50d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:27.940454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1a8d13-6a55-456a-9dea-30af1147c50d None admin] PATH_INFO: `/v3/projects/28e7a09459514ff78540606599ff1d26` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:27.941538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1a8d13-6a55-456a-9dea-30af1147c50d None admin] RBAC: Authorizing `identity:get_project(project_id=28e7a09459514ff78540606599ff1d26)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:27.942981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1a8d13-6a55-456a-9dea-30af1147c50d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:27.944581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 599/1191] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:19:27 2022] GET /identity/v3/projects/28e7a09459514ff78540606599ff1d26 => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:27.950825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:28.047727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-131bfcc4-503b-43cd-b9d0-1468a787c5f9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:28.072767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:28.081035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e216e2-56ba-4775-bdba-896cb54701d3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:28.081680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e216e2-56ba-4775-bdba-896cb54701d3 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:28.081952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e216e2-56ba-4775-bdba-896cb54701d3 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:28.082219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e216e2-56ba-4775-bdba-896cb54701d3 None admin] PATH_INFO: `/v3/projects/28e7a09459514ff78540606599ff1d26/users/db742fe9329d4169969174910605dc94/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:28.084149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e216e2-56ba-4775-bdba-896cb54701d3 None admin] RBAC: Authorizing `identity:create_grant(project_id=28e7a09459514ff78540606599ff1d26, user_id=db742fe9329d4169969174910605dc94, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:28.085557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e216e2-56ba-4775-bdba-896cb54701d3 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:28.156610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 593/1192] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:19:27 2022] PUT /identity/v3/projects/28e7a09459514ff78540606599ff1d26/users/db742fe9329d4169969174910605dc94/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:19:30.273635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdb112d-d4ec-4e03-9c8b-0747b98ba13c None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:30.274196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdb112d-d4ec-4e03-9c8b-0747b98ba13c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:30.274196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acdb112d-d4ec-4e03-9c8b-0747b98ba13c None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:30.275534 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 600/1193] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:19:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:19:30.280852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:30.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:30.281405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:30.300374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:19:30.301592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:19:30.367411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 594/1194] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:19:30 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:19:30.636172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdb112d-d4ec-4e03-9c8b-0747b98ba13c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:30.663233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:30.734078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acdb112d-d4ec-4e03-9c8b-0747b98ba13c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:30.767293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:30.767982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:30.768256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:30.768524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:30.778541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:19:30.783388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:19:30.784501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:19:30.785631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 601/1195] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:19:30 2022] GET /identity/v3/roles/admin => generated 83 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:19:30.790451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:30.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa614fb-8808-4614-b405-44106e7f1ff5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:30.901034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:30.918257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:30.918865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:30.919143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:30.919407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:30.920048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:30.920345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:30.921682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:30.930147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 595/1196] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:19:30 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:30.936236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:31.017167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:31.031117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa46942d-c1b4-491b-8429-fed0acd924bd None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:31.063578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:31.065964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:31.066236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:31.066503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] PATH_INFO: `/v3/users/db742fe9329d4169969174910605dc94` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:31.067662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] RBAC: Authorizing `identity:get_user(user_id=db742fe9329d4169969174910605dc94)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:31.069079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:31.070648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 602/1197] 10.222.0.64 () {56 vars in 1177 bytes} [Tue Sep 27 16:19:30 2022] GET /identity/v3/users/db742fe9329d4169969174910605dc94 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:31.076676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:31.174120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b5041f-6c2a-41b6-8d18-c9e76d5502e0 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:31.207010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:31.207596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:31.207873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:31.208144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:31.254191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:31.275177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:31.276696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:31.372564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 596/1198] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:19:31 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:19:31.378681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:31.476907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93367bf-09d1-46a5-88ad-ce928097bbb0 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:31.508347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03ac18b-1523-4044-aed8-19412751c584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:31.508956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03ac18b-1523-4044-aed8-19412751c584 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:31.509233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03ac18b-1523-4044-aed8-19412751c584 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:31.509507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03ac18b-1523-4044-aed8-19412751c584 None admin] PATH_INFO: `/v3/projects/28e7a09459514ff78540606599ff1d26` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:31.510596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03ac18b-1523-4044-aed8-19412751c584 None admin] RBAC: Authorizing `identity:get_project(project_id=28e7a09459514ff78540606599ff1d26)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:31.512040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03ac18b-1523-4044-aed8-19412751c584 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:31.513634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 603/1199] 10.222.0.64 () {56 vars in 1186 bytes} [Tue Sep 27 16:19:31 2022] GET /identity/v3/projects/28e7a09459514ff78540606599ff1d26 => generated 361 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:31.520240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:19:31.595746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:31.616569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7a92b2-d017-423e-8e21-c27574df5408 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:19:31.651397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:19:31.651999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:19:31.652281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:19:31.652555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:19:31.653272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=28e7a09459514ff78540606599ff1d26, user.id=db742fe9329d4169969174910605dc94) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:19:31.653581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:19:31.655039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6c2699-ba40-4860-b54e-33ef52589d6c None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:19:31.660703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 597/1200] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:19:31 2022] GET /identity/v3/role_assignments?user.id=db742fe9329d4169969174910605dc94&scope.project.id=28e7a09459514ff78540606599ff1d26&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:19:41.504270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:19:41.658203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=29 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:44.675783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba78502-0dbe-4141-8848-7677e5d47e5b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:44.676399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba78502-0dbe-4141-8848-7677e5d47e5b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:44.676399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba78502-0dbe-4141-8848-7677e5d47e5b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:44.677743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 604/1201] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:20:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:20:44.683126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:44.683554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:44.683739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:44.703482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:20:44.704687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:20:44.775408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 598/1202] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:20:44 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:20:45.054718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba78502-0dbe-4141-8848-7677e5d47e5b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:45.156916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba78502-0dbe-4141-8848-7677e5d47e5b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:45.190598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:45.191238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:45.191520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:45.191790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:45.202985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:45.204431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:45.219340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:45.226132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:45.227159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:45.227159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:45.227159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:20:45.227159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 605/1203] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:45.227417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:45.325079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1700e8b2-6ad4-4848-b49e-3a75727c0fc9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:45.357880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:45.363784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:45.364395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:45.364677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:45.364951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:45.365588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:45.365895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:45.367404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b59d0e44-8e5b-45e8-91c6-4484ec1734db None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:45.379062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 599/1204] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:45.384634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:45.438318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:45.485277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-368b68f6-37e5-46a1-8317-399ff05d4e06 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:45.517977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c680d9d6-6e5a-4e0d-a08b-d4649ba3755b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:45.518572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680d9d6-6e5a-4e0d-a08b-d4649ba3755b None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:45.518882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680d9d6-6e5a-4e0d-a08b-d4649ba3755b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:45.519155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c680d9d6-6e5a-4e0d-a08b-d4649ba3755b None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:45.519802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c680d9d6-6e5a-4e0d-a08b-d4649ba3755b None admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:45.521288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c680d9d6-6e5a-4e0d-a08b-d4649ba3755b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:45.606606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 606/1205] 10.222.0.64 () {60 vars in 1132 bytes} [Tue Sep 27 16:20:45 2022] POST /identity/v3/users => generated 251 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:20:47.788761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cdeb87-06b5-439a-af52-edd749de4b38 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:47.789326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cdeb87-06b5-439a-af52-edd749de4b38 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:47.789326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cdeb87-06b5-439a-af52-edd749de4b38 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:47.790666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 600/1206] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:20:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:20:47.796652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:47.797118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:47.797210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:47.820511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:20:47.821953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:20:47.887049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 607/1207] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:20:47 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:20:48.168336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cdeb87-06b5-439a-af52-edd749de4b38 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:48.274494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cdeb87-06b5-439a-af52-edd749de4b38 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:48.311113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:48.311750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:48.312031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:48.312305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:48.322844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:20:48.328387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:48.329541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:48.330421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 601/1208] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:20:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:48.335680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:48.375161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:48.432260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e4347-3128-4f66-9c6e-c9ec7d0e0785 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:48.467943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:48.468543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:48.468816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:48.469082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:48.469737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:48.470036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:48.471425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:48.479204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 608/1209] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:20:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:48.492499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:48.543585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:48.592642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a000b72a-5f62-4d94-8696-32778b9baf17 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:48.624766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:48.625384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:48.625663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:48.625934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:48.636489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:48.639614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:48.652395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:48.654101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:48.656110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:48.656110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:48.656110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:48.656110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 602/1210] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:48.660611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:48.761790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:48.764656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9414711-6df0-4e48-82e7-95edf34f3227 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:48.809647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:48.810334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:48.810514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:48.810806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:48.811426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:48.811724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:48.813077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:48.828783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 609/1211] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:48 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:48.834533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:48.933337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ee67c58-e34a-4b70-9a79-d0d63e905835 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:48.934864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:48.967656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:48.968278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:48.968556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:48.968828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] PATH_INFO: `/v3/users/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:48.975629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:48.977091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:20:48.989558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:48.990749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:48.991973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 603/1212] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:48 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:48.996620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:49.094428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d28a91-279f-493a-8b10-4ef99f01c288 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:49.127756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:49.128355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:49.128631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:49.128896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:49.129645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:49.129940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:49.131393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:49.149928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 610/1213] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Sep 27 16:20:48 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:49.156558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:49.256021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1a586a4-b7f6-4ab3-b012-7cbdbc951bee None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:49.289062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:49.289662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:49.289928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:49.290198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:49.319196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:49.357209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:49.358650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:49.458216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 604/1214] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:20:49 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:20:49.464921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:49.483254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:49.560121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92450ac2-e675-4aa3-ab87-a0dce7c75fd6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:49.594578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:49.595196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:49.595477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:49.595749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:49.605632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:49.607121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:49.621373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:49.622482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:49.623593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:49.623593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:49.623593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:20:49.623593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 611/1215] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:49 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:49.628494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:49.689317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:49.728121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1398574d-6dae-458f-a4ab-92e655b2d29d None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:49.763591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:49.764191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:49.764468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:49.764736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:49.765422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:49.765723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:49.767116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:49.778998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 605/1216] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:49.784742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:49.853851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:49.883936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664009c2-797e-4d3f-bdf0-6c23cea22fb4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:49.917182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:49.917793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:49.918071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:49.918454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:49.925719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:49.927186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:20:49.938945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:49.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:20:49.941119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 612/1217] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:20:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:49.946172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:50.042050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1fb34a4e-b85d-44f2-bcae-d505c8db4e9a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:50.065685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:50.080640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:50.081248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:50.081521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:50.081792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:50.082472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:50.082810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:50.084227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7f82f01-f47d-4abe-a422-11bf8e89fd94 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:50.097470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 606/1218] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:20:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:50.103620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:50.202486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae76822c-867c-4f7d-b485-8de1a0a4e6d2 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:50.229760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:50.235057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:50.235649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:50.235920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:50.236183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:50.236892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=4a39bfa348af4f499621fcb5d612e82c) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:50.237193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:50.238597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3058dd5c-d2ae-4ee0-ae24-0bbac9d7cbfc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:50.243755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 613/1219] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:20:50 2022] GET /identity/v3/role_assignments?user.id=4a39bfa348af4f499621fcb5d612e82c&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:52.378010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c7847c-907c-48c3-84c3-86b8214d7fe8 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:52.378520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c7847c-907c-48c3-84c3-86b8214d7fe8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:52.378615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2c7847c-907c-48c3-84c3-86b8214d7fe8 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:52.379960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 607/1220] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:20:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:20:52.384998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:52.385463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:52.385564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:52.403397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:20:52.404563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:20:52.469820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 614/1221] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:20:52 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:20:52.746938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c7847c-907c-48c3-84c3-86b8214d7fe8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:52.848649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2c7847c-907c-48c3-84c3-86b8214d7fe8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:52.882654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:52.883304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:52.883576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:52.883840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:52.894105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:20:52.902298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:52.903458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:52.904445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 608/1222] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:20:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:52.909885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.008151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3595dff-5e8b-4e0d-95d2-12097c904d61 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.031046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:53.041362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:53.042605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:53.042906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:53.043173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:53.043823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] RBAC: Adding query filter params (name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:53.044120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:53.045468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:53.053182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 615/1223] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:20:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:53.059253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.105125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:53.160981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc494475-05c6-4646-b5e4-89d01bbb9fb8 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.194522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:53.195194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:53.195469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:53.195991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:53.206596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:53.208703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:53.224810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:53.225962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:53.227090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:53.227090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:53.227090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:53.227090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 609/1224] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:53.231656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.330791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3398a057-8944-4f77-844e-c8a371997ee7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.365000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:53.365632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:53.365959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:53.366226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:53.366894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:53.367193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:53.368538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:53.379798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 616/1225] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:53.386031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.477976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:53.487511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c278198-a245-4e38-b3cc-6c9960e6807f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.524437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:53.525055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:53.525333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:53.525681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] PATH_INFO: `/v3/users/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:53.533083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:53.534542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:20:53.546246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:53.547433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:53.548623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 610/1226] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:53 2022] GET /identity/v3/users/placement => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:53.553220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.599294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:53.652457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec5d0ad-7314-4172-ad5f-54479f3240d7 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.685709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:53.686298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:53.686602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:53.686900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:53.687619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:53.687921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:53.689333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:53.706080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 617/1227] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Sep 27 16:20:53 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:53.711915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.811891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f88e64b-a390-4b6a-bf36-1c7e83bf979b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.847081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:53.847697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:53.847973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:53.848243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:53.859232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:53.860263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:53.875346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:53.876516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:53.877597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:53.877597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:53.877597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:53.877597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 611/1228] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:53.882495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:53.981117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa144e1-6fb1-4e36-8d1d-605bc26c3b40 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:53.983449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:54.015595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:54.016190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:54.016463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:54.016730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:54.017357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:54.017653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:54.019142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:54.030932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 618/1229] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:54.036614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:54.046323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:54.138254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6289c2f7-4189-46a3-894a-f3e31d78e49f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:54.172578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:54.173196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:54.173471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:54.173740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:54.180621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:54.182049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:20:54.193682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:54.194647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:54.199720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 612/1230] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:20:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:54.201794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:54.297284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3696dfb4-2d86-4ee1-a168-620ac3dea76b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:54.330709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:54.331329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:54.331604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:54.331892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:54.332572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:54.332876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:54.334285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b245415-c62c-4d5f-9ef2-c155c2425b30 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:54.348286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 619/1231] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:20:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:54.354420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:54.418521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:54.453800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d03956a-7108-46dc-870d-2fe3cc856221 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:54.485839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96c2138-a936-47b0-bb3d-598e30e03599 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:54.486479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c2138-a936-47b0-bb3d-598e30e03599 None admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:54.486760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c2138-a936-47b0-bb3d-598e30e03599 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:54.487062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96c2138-a936-47b0-bb3d-598e30e03599 None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/4a39bfa348af4f499621fcb5d612e82c/roles/6d99311b2a224f2fb55411c7e94b31d9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:54.501243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96c2138-a936-47b0-bb3d-598e30e03599 None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=4a39bfa348af4f499621fcb5d612e82c, role_id=6d99311b2a224f2fb55411c7e94b31d9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:54.502671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96c2138-a936-47b0-bb3d-598e30e03599 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:54.613552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 613/1232] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:20:54 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/4a39bfa348af4f499621fcb5d612e82c/roles/6d99311b2a224f2fb55411c7e94b31d9 => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:20:56.761109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b9841b-3598-4e2a-abb8-104442de9db2 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:56.761649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b9841b-3598-4e2a-abb8-104442de9db2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:56.761649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b9841b-3598-4e2a-abb8-104442de9db2 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:56.763032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 620/1233] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:20:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:20:56.768139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:56.768544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:56.768679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:56.787136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:20:56.799405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:20:56.854510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 614/1234] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:20:56 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:20:57.131592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b9841b-3598-4e2a-abb8-104442de9db2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:57.157175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:57.231839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b9841b-3598-4e2a-abb8-104442de9db2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:57.265278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:57.265977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:57.266207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:57.266476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] PATH_INFO: `/v3/roles/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:57.276241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:20:57.281267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:57.282183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:20:57.283077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 621/1235] 10.222.0.64 () {56 vars in 1102 bytes} [Tue Sep 27 16:20:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:57.288409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:57.335461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:57.387235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c89a6a1d-d05b-42fd-bedf-7d3cd905b529 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:57.422927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:57.423524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:57.423797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:57.424066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:57.424715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] RBAC: Adding query filter params (name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:57.425012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:57.426368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:57.435050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 615/1236] 10.222.0.64 () {56 vars in 1103 bytes} [Tue Sep 27 16:20:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:57.447334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:57.531358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:57.546449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9848d21e-ef96-4fb5-b781-d2d4ae381ba9 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:57.580158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:57.580776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:57.581053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:57.581322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:57.591100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:57.592522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:57.607641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:57.608621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:57.609716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:57.609716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:57.609716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:20:57.609716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 622/1237] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:57.614676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:57.715122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e00daab2-1784-433a-8a64-0f512b2325c9 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:57.726600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:57.752056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:57.752650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:57.752925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:57.753193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:57.753816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:57.754113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:57.755496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:57.767173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 616/1238] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:57.772927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:57.872314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd5a4c-f56c-4a43-b5b1-febd968e7735 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:57.905416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:57.906184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:57.906466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:57.906734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:57.909404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:57.915239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:57.916768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:20:57.928748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:57.929790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:20:57.930755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 623/1239] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:57 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:57.936360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:58.036563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d82067d-1222-4a3d-b04e-3efb214c6b20 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:58.070570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:58.071454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:58.071732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:58.072028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:58.072769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:58.073069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:58.074503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:58.093156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 617/1240] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Sep 27 16:20:57 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:58.099002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:58.196481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0fc65bc-339b-44df-a5ad-a41276aac461 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:58.229046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:58.230333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:58.230821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:58.231096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:58.292727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:58.299106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:58.300556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:58.396967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 624/1241] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:20:58 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 298 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:20:58.403658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:58.430503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:58.502002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23faca7d-ee4e-4225-9715-2b3804cf1c0f None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:58.538071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:58.538696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:58.539002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:58.539271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:58.549703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:58.551124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:20:58.565291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:58.566389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:20:58.567708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:20:58.567708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:20:58.567708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:58.567708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 618/1242] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:20:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:20:58.572463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:58.662921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:58.668474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7b10d4-d633-4914-a724-84fef152d7fa None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:58.701725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:58.702328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:58.702608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:58.702911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:58.703550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:58.703855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:58.705272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:58.716692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 625/1243] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:20:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:20:58.722368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:58.799619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:20:58.820303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37edfe8a-67dc-40e2-96cf-8138c3d947cc None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:58.852650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:58.853273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:58.853556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:58.853830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:58.860866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:58.862311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:20:58.873478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:20:58.881354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:20:58.882505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 619/1244] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:20:58 2022] GET /identity/v3/projects/service => generated 88 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:20:58.883222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:58.977866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17af66eb-90ce-42fa-a22d-b1e66eaa925c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:59.009767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:59.010355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:59.010645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:59.010948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:59.011625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:59.011925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:59.013331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c482b69b-f654-4a1e-b437-1c90d0d85026 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:59.024695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 626/1245] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:20:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:20:59.031576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:20:59.130499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95b7c5e8-c5bf-415a-b792-4456473c46a7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:20:59.164140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:20:59.164738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:20:59.165010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:20:59.165279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:20:59.165988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] RBAC: Adding query filter params (role.id=6d99311b2a224f2fb55411c7e94b31d9, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=4a39bfa348af4f499621fcb5d612e82c) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:20:59.166291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:20:59.167750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f407c764-a9f1-47bf-99c0-d1543d037a29 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:20:59.173255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 620/1246] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:20:59 2022] GET /identity/v3/role_assignments?user.id=4a39bfa348af4f499621fcb5d612e82c&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=6d99311b2a224f2fb55411c7e94b31d9 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:01.403489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ff324e-6a0a-4ced-971c-c6b858718b0f None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:01.404056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ff324e-6a0a-4ced-971c-c6b858718b0f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:01.404056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40ff324e-6a0a-4ced-971c-c6b858718b0f None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:01.405363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 627/1247] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:01.410726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:01.411153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:01.411296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:01.418612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:01.431012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:01.432199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:01.500826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 621/1248] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:01 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:01.773564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ff324e-6a0a-4ced-971c-c6b858718b0f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:01.791555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:01.877478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40ff324e-6a0a-4ced-971c-c6b858718b0f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:01.913408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:01.914030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:01.914311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:01.914580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:01.924824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:21:01.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:01.930799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:01.932093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 628/1249] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:21:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:01.937571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:02.034011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61881703-7088-4f8d-af7d-32eb3b13111b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:02.063336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:02.068070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:02.068664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:02.068938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:02.069211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:02.069885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:02.070207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:02.071857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:02.080336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 622/1250] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:21:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:02.086230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:02.163669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:02.185730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0f9dcc9-521f-4373-8dfc-15865ba3f7f1 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:02.217618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:02.218232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:02.218508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:02.218797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:02.228672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:02.230259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:21:02.243747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:02.244892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:21:02.246694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:21:02.246694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:02.246694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:02.246694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 629/1251] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:02.251245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:02.351486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1b91fdc-92a2-4071-bd62-9a09bd32c01a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:02.384930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:02.385527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:02.385880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:02.386667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:02.387322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:02.387626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:02.389048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:02.400538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 623/1252] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:21:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:02.406330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:02.504941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7481d6-8f03-41b0-961f-88812e775524 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:02.530233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:02.538015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:02.543427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:02.543427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:02.543427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:02.547254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:02.548683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:21:02.560222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:02.561427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:02.562605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 630/1253] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:02.567421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:02.624203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:02.666923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a8fba88-8a59-49b1-8657-809ac7f75d45 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:02.700518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:02.701124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:02.701402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:02.701670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:02.702382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:02.702685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:02.704304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:02.721486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 624/1254] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Sep 27 16:21:02 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:02.726852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:02.824863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba99a942-b341-4b06-837d-9c874e372cb4 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:02.858313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde78d54-c148-4943-897d-000252c64acc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:02.858969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde78d54-c148-4943-897d-000252c64acc None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:02.859255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde78d54-c148-4943-897d-000252c64acc None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:02.859517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde78d54-c148-4943-897d-000252c64acc None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:02.913881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:02.926192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde78d54-c148-4943-897d-000252c64acc None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:02.927658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde78d54-c148-4943-897d-000252c64acc None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:03.020612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 631/1255] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:21:02 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:21:03.027255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:03.123828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b739c05-83de-4d1f-b1f8-b0f993885875 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:03.136434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:03.157261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:03.157879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:03.158152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:03.158418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:03.168872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:03.170252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:21:03.185346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:03.186302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:21:03.187309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:21:03.187309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:03.187309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:03.187309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 625/1256] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:03.192536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde78d54-c148-4943-897d-000252c64acc None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:03.280710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:03.292788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde78d54-c148-4943-897d-000252c64acc None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:03.327242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:03.327847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:03.328129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:03.328415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:03.329053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:03.329377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:03.330734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:03.342750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 632/1257] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:21:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:03.348599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:03.448691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5e96ac4-0dec-4e58-a828-d05cd29be594 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:03.484789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:03.485412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:03.485687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:03.485952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] PATH_INFO: `/v3/projects/service` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:03.492567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:03.494007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:21:03.505762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:03.507526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:03.508629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 626/1258] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:21:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:03.512876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:03.611999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33a0f839-1731-4e9f-bedc-ab23550af922 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:03.643506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:03.644099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:03.644373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:03.644638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:03.645312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:03.645607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:03.647052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b635216e-8fbd-4d0d-bce8-3e3f865c7ee0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:03.659363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 633/1259] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:21:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:03.665419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:03.690503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:03.769419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6406bf3c-54d5-44ec-aaf2-4e2af40044ed None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:03.804840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:03.805439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:03.805717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:03.805986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:03.806823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=4a39bfa348af4f499621fcb5d612e82c) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:03.807135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:03.810024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0b49b84-e045-4f3e-97ad-2fe4a7a961e0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:03.815492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 627/1260] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:21:03 2022] GET /identity/v3/role_assignments?user.id=4a39bfa348af4f499621fcb5d612e82c&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 259 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:06.050465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c18ec-25af-40e3-b06b-c196201d3820 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:06.051087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c18ec-25af-40e3-b06b-c196201d3820 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:06.051087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-969c18ec-25af-40e3-b06b-c196201d3820 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:06.052373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 634/1261] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:06.057539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:06.058472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:06.058606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:06.077668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:06.078961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:06.144145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 628/1262] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:06 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:06.418571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c18ec-25af-40e3-b06b-c196201d3820 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:06.429547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:06.516856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-969c18ec-25af-40e3-b06b-c196201d3820 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:06.551017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:06.551639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:06.551914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:06.552259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:06.563054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:21:06.567997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:06.569171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:21:06.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:06.574641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 635/1263] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:21:06 2022] GET /identity/v3/roles/admin => generated 83 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:06.575712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:06.594547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:06.673432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f523d1e-ace2-4ade-a299-18c429f4a668 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:06.706646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:06.707282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:06.707563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:06.707836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:06.708496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:06.708799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:06.710164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:06.719158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 629/1264] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:21:06 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:06.725039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:06.790866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:06.825709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd83f46-df25-4eec-8c4b-d71d8e478b6c None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:06.858625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:06.859263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:06.859536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:06.859803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:06.869233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:06.870661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:21:06.885572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:06.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:21:06.897205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:21:06.897205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:06.897205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:06.897205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 636/1265] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:06 2022] GET /identity/v3/domains/Default => generated 87 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Sep 27 16:21:06.897467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:06.974689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:06.993178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1177a279-2a93-4f32-8b5c-01111f521761 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:07.027154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:07.028155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:07.028666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:07.028947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:07.029581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:07.029886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:07.031271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:07.043160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 630/1266] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:21:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:07.048896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:07.147694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c82ffa-84a3-41d4-9fac-bb30a4ddcdc5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:07.166354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:07.180283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:07.180898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:07.181171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:07.181440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] PATH_INFO: `/v3/users/placement` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:07.189373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:07.190818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:21:07.202327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:07.203577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:07.204754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 637/1267] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:07 2022] GET /identity/v3/users/placement => generated 87 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:07.209234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:07.314320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d35cf132-d1bb-405c-aec4-84abc92aa894 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:07.348393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:07.348999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:07.349275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:07.349547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:07.350265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:07.350686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:07.352140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:07.361837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:07.369638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 631/1268] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Sep 27 16:21:07 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:07.375756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:07.475299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264ca1da-ec29-4b92-a7a7-a9026729940f None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:07.507715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:07.508324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:07.508602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:07.508867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:07.519214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:07.520762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:21:07.535348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:07.536469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:21:07.537585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:21:07.537585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:07.537585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:07.537585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 638/1269] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:07 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:07.543215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:07.641832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69fbbc0f-1e22-4157-ba1a-f1241f9d9b08 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:07.674897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:07.675506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:07.675784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:07.676057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:07.676683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:07.676985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:07.678350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:07.689995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 632/1270] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:21:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:07.695944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:07.727058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:07.792840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74faacd-2727-4a71-83c3-3efee07b9854 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:07.831574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:07.832245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:07.832527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:07.832792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:07.839908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:07.841340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:21:07.852710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:07.853820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:07.855380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 639/1271] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:21:07 2022] GET /identity/v3/projects/service => generated 88 bytes in 159 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:07.860796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:07.923187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:07.965720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e27920c-0e79-4b7e-b110-e17a4c08b854 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:08.006248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:08.006762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:08.007066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:08.007338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:08.008029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:08.008338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:08.009766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-254dc24d-a561-4915-b81c-24b0309c1498 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:08.024914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 633/1272] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:21:07 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:08.031113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:08.124243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:08.132683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac500a27-032b-4201-9a44-4a5425f5338b None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:08.164336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f0e6d36-560a-4ee8-8fdf-9241e5b294f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:08.164982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0e6d36-560a-4ee8-8fdf-9241e5b294f0 None admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:08.165243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0e6d36-560a-4ee8-8fdf-9241e5b294f0 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:08.165510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f0e6d36-560a-4ee8-8fdf-9241e5b294f0 None admin] PATH_INFO: `/v3/projects/6519177f56a241148d3389f111fb8b28/users/4a39bfa348af4f499621fcb5d612e82c/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:08.167964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0e6d36-560a-4ee8-8fdf-9241e5b294f0 None admin] RBAC: Authorizing `identity:create_grant(project_id=6519177f56a241148d3389f111fb8b28, user_id=4a39bfa348af4f499621fcb5d612e82c, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:08.169348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f0e6d36-560a-4ee8-8fdf-9241e5b294f0 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:08.263980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 640/1273] 10.222.0.64 () {58 vars in 1439 bytes} [Tue Sep 27 16:21:08 2022] PUT /identity/v3/projects/6519177f56a241148d3389f111fb8b28/users/4a39bfa348af4f499621fcb5d612e82c/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:21:10.401271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888d0555-823e-4fd7-abdf-029c2f449309 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:10.402398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888d0555-823e-4fd7-abdf-029c2f449309 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:10.402398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-888d0555-823e-4fd7-abdf-029c2f449309 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:10.403689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 634/1274] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:10.408854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:10.409256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:10.409469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:10.427839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:10.429022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:10.496949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 641/1275] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:10 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:10.784465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888d0555-823e-4fd7-abdf-029c2f449309 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:10.883897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-888d0555-823e-4fd7-abdf-029c2f449309 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:10.901235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:10.917163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:10.917777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:10.918050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:10.918315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:10.928125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(role_id) Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Sep 27 16:21:10.933535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:10.934766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:10.936046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 635/1276] 10.222.0.64 () {56 vars in 1096 bytes} [Tue Sep 27 16:21:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 151 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:10.940815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:11.019951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:11.043727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d8919a-f8bb-4cf7-b558-3bad509df7a6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:11.069978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:11.070856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:11.074163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:11.074163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:11.074163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] RBAC: Adding query filter params (name=admin) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:11.074163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:11.074613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:11.082013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 642/1277] 10.222.0.64 () {56 vars in 1099 bytes} [Tue Sep 27 16:21:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:11.087908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:11.187472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae02fbc-952e-4685-b26e-2869957919ba None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:11.221828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:11.222449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:11.222725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:11.223025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:11.233501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:11.234986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:21:11.249791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:11.250976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:21:11.252074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:21:11.252074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:11.252074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:11.252074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 636/1278] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:11.256778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:11.355885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de58a61c-9a9f-4f70-8748-87696432a481 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:11.388144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:11.388735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:11.389052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:11.389332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:11.389948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:11.390244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:11.391656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:11.402828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 643/1279] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:21:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:11.408583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:11.439952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:11.509201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4edfcb34-672e-4918-97cc-ddb3cde9b671 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:11.542681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:11.543498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:11.543775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:11.544041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] PATH_INFO: `/v3/users/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:11.550818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:11.552252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_user(user_id) Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Sep 27 16:21:11.563743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:11.564847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:11.566035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 637/1280] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:11 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:11.571072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:11.574985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:11.668082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf24ddef-2f66-4c3c-b313-ffa0c49075f5 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:11.700494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:11.701084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:11.701356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:11.701620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:11.702325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:11.702637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:11.704069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:11.720922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 644/1281] 10.222.0.64 () {56 vars in 1143 bytes} [Tue Sep 27 16:21:11 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:21:11.726946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:11.815642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:11.824934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9c7238d-0fe0-4676-a58d-a9cda6165802 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:11.858539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:11.859169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:11.859447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:11.859717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:11.925099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:11.926476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:12.012089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:12.022152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 638/1282] 10.222.0.64 () {58 vars in 1282 bytes} [Tue Sep 27 16:21:11 2022] GET /identity/v3/auth/tokens => generated 2118 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Sep 27 16:21:12.028618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:12.087992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:12.124754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e546f854-713d-4e84-a801-4b0b9fb7e8cb None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:12.158355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:12.158999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:12.159279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:12.159550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:12.169193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:12.170624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Sep 27 16:21:12.184695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:12.185780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Sep 27 16:21:12.186858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Sep 27 16:21:12.186858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Sep 27 16:21:12.186858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:12.186858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 645/1283] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:21:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:12.191626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:12.293475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f3e0ff5-9ea7-46b6-bcbe-12a74f9ed0b7 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:12.327448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:12.328045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:12.328321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:12.328587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:12.329215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:12.329511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:12.330905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:12.342252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 639/1284] 10.222.0.64 () {56 vars in 1109 bytes} [Tue Sep 27 16:21:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:12.347945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:12.448011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b25594e-3ccc-4dbc-94ce-f3973604cad6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:12.461926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:12.481523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:12.482150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:12.482440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:12.482716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] PATH_INFO: `/v3/projects/service` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:12.489125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:12.490556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(project_id) Sep 27 16:21:12.502759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:12.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: ERROR keystone.server.flask.application Sep 27 16:21:12.505017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 646/1285] 10.222.0.64 () {56 vars in 1111 bytes} [Tue Sep 27 16:21:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:12.509765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:12.559671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:12.612745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-672459cf-db0e-48be-80a0-e731f278e20b None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:12.645792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:12.646394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:12.646671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:12.646973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:12.647655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:12.647959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:12.649368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-959e75cb-6c68-47fa-afe0-20f3bbf07df9 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:12.661007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 640/1286] 10.222.0.64 () {56 vars in 1148 bytes} [Tue Sep 27 16:21:12 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:12.667699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:12.774883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b4cbc2-6e4d-4082-9d1a-bbc421f497b6 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:12.810647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:12.811300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:12.811581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:12.811849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:12.812582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] RBAC: Adding query filter params (role.id=bd6f5e4de7d6438c962970b4fda1866a, scope.project.id=6519177f56a241148d3389f111fb8b28, user.id=4a39bfa348af4f499621fcb5d612e82c) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:12.812887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:12.814344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b89f8b5-52ca-4bd3-8125-15df765d6c79 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:12.821196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 647/1287] 10.222.0.64 () {56 vars in 1374 bytes} [Tue Sep 27 16:21:12 2022] GET /identity/v3/role_assignments?user.id=4a39bfa348af4f499621fcb5d612e82c&scope.project.id=6519177f56a241148d3389f111fb8b28&role.id=bd6f5e4de7d6438c962970b4fda1866a => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:15.020107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba6daad-e0d0-4402-97da-5fc5416562b2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:15.020671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba6daad-e0d0-4402-97da-5fc5416562b2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:15.020778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba6daad-e0d0-4402-97da-5fc5416562b2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:15.022036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 641/1288] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:15.027232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:15.027625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:15.027781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:15.046260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:15.047478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:15.076516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:15.112366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 648/1289] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:15 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:15.379420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba6daad-e0d0-4402-97da-5fc5416562b2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:15.480829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba6daad-e0d0-4402-97da-5fc5416562b2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:15.491451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:15.514112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:15.514794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:15.515076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:15.515346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] PATH_INFO: `/v3/services/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:15.515920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:15.517278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:15.526021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:15.527855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:15.529124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 642/1290] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:21:15 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:15.533899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:15.629769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88362ff8-a154-44bf-bc55-a178088e4d0a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:15.676664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:15.677299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:15.677544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:15.677811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:15.678439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:15.678738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:15.680119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b63170-8897-41ef-9705-5806dd76cff4 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:15.685710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 649/1291] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Sep 27 16:21:15 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:21:15.691449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:15.788748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66c57c5d-15ce-462a-b376-6fc4c6fd912a None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:15.822597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:15.823335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:15.823622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:15.823910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] PATH_INFO: `/v3/services` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:15.824522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] RBAC: Adding query filter params (type=placement) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:15.824821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:15.826280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8e6582b-a91f-42c8-b529-58b066e661ee None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:15.830729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 643/1292] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Sep 27 16:21:15 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:17.979791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63d399b-2cdf-4fd7-bea7-88c674207477 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:17.980329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63d399b-2cdf-4fd7-bea7-88c674207477 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:17.980329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b63d399b-2cdf-4fd7-bea7-88c674207477 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:17.981663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 650/1293] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:17.986769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db2ab7-6113-4032-bb00-acca68a1f0c6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:17.987195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db2ab7-6113-4032-bb00-acca68a1f0c6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:17.987340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4db2ab7-6113-4032-bb00-acca68a1f0c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:18.005513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f4db2ab7-6113-4032-bb00-acca68a1f0c6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:18.006678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f4db2ab7-6113-4032-bb00-acca68a1f0c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:18.036535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:18.072679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 644/1294] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:17 2022] POST /identity/v3/auth/tokens => generated 2118 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:18.344257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63d399b-2cdf-4fd7-bea7-88c674207477 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:18.375214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:18.444824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b63d399b-2cdf-4fd7-bea7-88c674207477 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:18.479212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c16d4f84-0937-44c0-ad71-e939ee30c09d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:18.480251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16d4f84-0937-44c0-ad71-e939ee30c09d None admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:18.480531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16d4f84-0937-44c0-ad71-e939ee30c09d None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:18.480807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c16d4f84-0937-44c0-ad71-e939ee30c09d None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:18.481462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16d4f84-0937-44c0-ad71-e939ee30c09d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:18.482845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c16d4f84-0937-44c0-ad71-e939ee30c09d None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:18.554609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 651/1295] 10.222.0.64 () {60 vars in 1141 bytes} [Tue Sep 27 16:21:18 2022] POST /identity/v3/services => generated 246 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:21:20.790937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b9123-b3a4-4838-9c6a-0a801b919b06 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:20.791466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b9123-b3a4-4838-9c6a-0a801b919b06 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:20.791559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-543b9123-b3a4-4838-9c6a-0a801b919b06 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:20.793075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 645/1296] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:20.798484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:20.798958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:20.799058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:20.817746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:20.818961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:20.896732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 652/1297] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:20 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:21.174750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b9123-b3a4-4838-9c6a-0a801b919b06 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:21.275033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-543b9123-b3a4-4838-9c6a-0a801b919b06 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:21.309510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:21.310130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:21.310404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:21.310672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:21.311339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:21.312696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:21.321505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:21.326985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:21.328019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 646/1298] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:21:21 2022] GET /identity/v3/services/placement => generated 90 bytes in 148 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:21.329126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:21.349851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:21.428732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b95bbd8-2139-4f30-8530-3046bcd60f86 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:21.461027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:21.461617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:21.461887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:21.462151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:21.462766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] RBAC: Adding query filter params (name=placement) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:21.463089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:21.465125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:21.469955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 653/1299] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Sep 27 16:21:21 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:21:21.476011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:21.508019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:21.578294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d10300d-8f5c-4b73-b835-ef53b78ce5a4 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:21.612114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:21.612712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:21.612988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:21.613257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:21.613948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] RBAC: Adding query filter params (interface=public, service_id=beeb9d228bad42f48589cb44b745e3c7, region_id=RegionOne) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:21.614253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:21.615652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46aec2a7-0617-414c-baa0-145d8b5a02e0 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:21.620690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 647/1300] 10.222.0.64 () {56 vars in 1251 bytes} [Tue Sep 27 16:21:21 2022] GET /identity/v3/endpoints?service_id=beeb9d228bad42f48589cb44b745e3c7&interface=public®ion_id=RegionOne => generated 194 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:21:21.626506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:21.706770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:21.724518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc972d6e-e19a-41ca-aa18-de4393d64b49 None admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:21.756966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-020e5757-935a-4d85-81dc-c1d0f7c836b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:21.757564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e5757-935a-4d85-81dc-c1d0f7c836b5 None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:21.757842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e5757-935a-4d85-81dc-c1d0f7c836b5 None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:21.758114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-020e5757-935a-4d85-81dc-c1d0f7c836b5 None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:21.758707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020e5757-935a-4d85-81dc-c1d0f7c836b5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:21.760086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-020e5757-935a-4d85-81dc-c1d0f7c836b5 None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:21.767491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 654/1301] 10.222.0.64 () {56 vars in 1087 bytes} [Tue Sep 27 16:21:21 2022] GET /identity/v3/services => generated 2191 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:21:23.921232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4178c206-9e6f-4339-ba1e-8973aa95abd1 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:23.921791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4178c206-9e6f-4339-ba1e-8973aa95abd1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:23.921791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4178c206-9e6f-4339-ba1e-8973aa95abd1 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:23.923192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 648/1302] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:21:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:21:23.928431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:23.928846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:23.928974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:23.947582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:21:23.952452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:21:24.013808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 655/1303] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:21:23 2022] POST /identity/v3/auth/tokens => generated 2221 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Sep 27 16:21:24.282443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4178c206-9e6f-4339-ba1e-8973aa95abd1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:24.377621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:24.384201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4178c206-9e6f-4339-ba1e-8973aa95abd1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:24.417284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:24.417904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:24.418538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:24.418538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] PATH_INFO: `/v3/services/placement` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:24.419135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:24.420625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Traceback (most recent call last): Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1820, in full_dispatch_request Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application rv = self.dispatch_request() Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1796, in dispatch_request Sep 27 16:21:24.429836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 107, in view Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._get_service(service_id) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Sep 27 16:21:24.431047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.get_or_create( Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application with Lock( Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self._enter() Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application return self.creator() Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application created_value = creator( Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: ERROR keystone.server.flask.application Sep 27 16:21:24.432070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 649/1304] 10.222.0.64 () {56 vars in 1117 bytes} [Tue Sep 27 16:21:24 2022] GET /identity/v3/services/placement => generated 90 bytes in 149 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Sep 27 16:21:24.437604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:24.513776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:24.535466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d72789c6-7c42-480d-84db-7c3f7e3dccaf None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:24.568624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:24.569216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:24.569493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:24.569760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] PATH_INFO: `/v3/services` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:24.570651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] RBAC: Adding query filter params (name=placement) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:21:24.570976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:24.576449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde0819a-e867-4e75-bd99-417a901ba51f None admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:24.578024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 656/1305] 10.222.0.64 () {56 vars in 1116 bytes} [Tue Sep 27 16:21:24 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:21:24.583920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:21:24.683497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-613689e2-569a-41c2-ae8a-e10aca180062 None admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:21:24.716479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c395f225-00de-4d18-929a-bee318b4176f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:21:24.717077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395f225-00de-4d18-929a-bee318b4176f None admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:21:24.717356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395f225-00de-4d18-929a-bee318b4176f None admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:21:24.717626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c395f225-00de-4d18-929a-bee318b4176f None admin] PATH_INFO: `/v3/endpoints` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:21:24.718292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c395f225-00de-4d18-929a-bee318b4176f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:21:24.719737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c395f225-00de-4d18-929a-bee318b4176f None admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:21:24.825885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 650/1306] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:21:24 2022] POST /identity/v3/endpoints => generated 331 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:21:34.575114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=30 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:21:34.821068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:37.918245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27034693-42a4-4311-855a-e25febe73019 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:37.918718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27034693-42a4-4311-855a-e25febe73019 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:37.918961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27034693-42a4-4311-855a-e25febe73019 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:37.920330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 657/1307] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:37.926455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219373a7-f217-4973-becd-2ecb6626293b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:37.928020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219373a7-f217-4973-becd-2ecb6626293b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:37.929706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-219373a7-f217-4973-becd-2ecb6626293b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:38.015474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-219373a7-f217-4973-becd-2ecb6626293b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:38.016718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-219373a7-f217-4973-becd-2ecb6626293b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:38.134346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 651/1308] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:27:38.435934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e5cd6-68f3-47d3-8816-cba09f012b6d None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:38.436214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e5cd6-68f3-47d3-8816-cba09f012b6d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:38.436637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a5e5cd6-68f3-47d3-8816-cba09f012b6d None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:38.438082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 658/1309] 10.222.0.64 () {54 vars in 964 bytes} [Tue Sep 27 16:27:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:38.444559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c30a72-b2e6-4ef8-ac44-e862e0871b7d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:38.445183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c30a72-b2e6-4ef8-ac44-e862e0871b7d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:38.445579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c30a72-b2e6-4ef8-ac44-e862e0871b7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:38.533085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c1c30a72-b2e6-4ef8-ac44-e862e0871b7d None None] MFA Rules not processed for user `28f0f9fef2f54e56a7a92d077c25d2b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:38.534900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c1c30a72-b2e6-4ef8-ac44-e862e0871b7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:38.560260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:38.617953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 652/1310] 10.222.0.64 () {58 vars in 1061 bytes} [Tue Sep 27 16:27:38 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:27:38.626997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e5cd6-68f3-47d3-8816-cba09f012b6d None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:38.628873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2a5e5cd6-68f3-47d3-8816-cba09f012b6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:38.748604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a5e5cd6-68f3-47d3-8816-cba09f012b6d None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:38.830896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:38.832135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:38.832490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:38.832834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:38.834544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:38.894909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:38.955537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:38.957082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69670db-5efc-4aa4-ac97-134a8b7ae071 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:38.979387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 659/1311] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 353 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:27:41.497583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21002acf-f4df-49c2-a9a0-87332ef3ca9b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:41.498088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21002acf-f4df-49c2-a9a0-87332ef3ca9b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:41.498088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21002acf-f4df-49c2-a9a0-87332ef3ca9b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:41.499436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 653/1312] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:41.504567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153e5b92-8597-4dfd-a591-b72ca7416ead None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:41.505013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153e5b92-8597-4dfd-a591-b72ca7416ead None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:41.505111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-153e5b92-8597-4dfd-a591-b72ca7416ead None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:41.586567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-153e5b92-8597-4dfd-a591-b72ca7416ead None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:41.587898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-153e5b92-8597-4dfd-a591-b72ca7416ead None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:41.686026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 660/1313] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:27:41.969637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21002acf-f4df-49c2-a9a0-87332ef3ca9b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:41.971205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-21002acf-f4df-49c2-a9a0-87332ef3ca9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:42.078536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21002acf-f4df-49c2-a9a0-87332ef3ca9b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:42.152440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:42.154737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:42.155363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:42.155645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:42.155913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:42.157540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:42.266064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:42.267464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-626223dd-f242-4575-aae7-e3cabe24329a service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:42.285586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 654/1314] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:27:45.048783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc88098-6921-4950-8e13-f0e617464fcc None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:45.049363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc88098-6921-4950-8e13-f0e617464fcc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:45.049363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc88098-6921-4950-8e13-f0e617464fcc None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:45.050670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 661/1315] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:45.056011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113cd7e6-6eeb-4aba-bb92-e11d6a75d8fb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:45.056427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113cd7e6-6eeb-4aba-bb92-e11d6a75d8fb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:45.056566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-113cd7e6-6eeb-4aba-bb92-e11d6a75d8fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:45.121893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-113cd7e6-6eeb-4aba-bb92-e11d6a75d8fb None None] MFA Rules not processed for user `e6f8c1e119d74e408654eed7e007c7f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:45.123131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-113cd7e6-6eeb-4aba-bb92-e11d6a75d8fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:45.174033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:45.224246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 655/1316] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:45 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:27:45.511782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc88098-6921-4950-8e13-f0e617464fcc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:45.513367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8fc88098-6921-4950-8e13-f0e617464fcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:45.523598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:45.620835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc88098-6921-4950-8e13-f0e617464fcc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:45.695346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:45.695944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:45.696218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:45.696484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:45.698060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:45.758792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:45.804163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:45.805944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a49753c-b961-4822-8310-fe278569f6e2 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:45.825712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 662/1317] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:45 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:27:48.807872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2718e-12ae-434a-b72f-c5f3e9115bcc None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:48.808444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2718e-12ae-434a-b72f-c5f3e9115bcc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:48.808444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2718e-12ae-434a-b72f-c5f3e9115bcc None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:48.809989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 656/1318] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:48.815139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f36e6-a419-424b-8537-54b322d0db9b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:48.815532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f36e6-a419-424b-8537-54b322d0db9b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:48.815673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f36e6-a419-424b-8537-54b322d0db9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:48.880987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-bf0f36e6-a419-424b-8537-54b322d0db9b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:48.882374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bf0f36e6-a419-424b-8537-54b322d0db9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:48.975409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 663/1319] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:48 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:27:49.262336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2718e-12ae-434a-b72f-c5f3e9115bcc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:49.263946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a3e2718e-12ae-434a-b72f-c5f3e9115bcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:49.370476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2718e-12ae-434a-b72f-c5f3e9115bcc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:49.441862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:49.442452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:49.442723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:49.443017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:49.444590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:49.451385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:49.552507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:49.553859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e157200-345e-4e05-99fd-d0343638a62f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:49.572308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 657/1320] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:49 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:27:53.004669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb57321-f68d-453a-adda-9395daa9d023 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:53.005280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb57321-f68d-453a-adda-9395daa9d023 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:53.005280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb57321-f68d-453a-adda-9395daa9d023 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:53.006574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 664/1321] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:53.012017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba03d1-3ee0-49c3-a0b6-f9ce3e96bc01 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:53.012484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba03d1-3ee0-49c3-a0b6-f9ce3e96bc01 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:53.012592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8ba03d1-3ee0-49c3-a0b6-f9ce3e96bc01 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:53.084249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d8ba03d1-3ee0-49c3-a0b6-f9ce3e96bc01 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:53.086319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d8ba03d1-3ee0-49c3-a0b6-f9ce3e96bc01 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:53.149787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:53.191435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 658/1322] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:27:53.475265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb57321-f68d-453a-adda-9395daa9d023 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:53.478799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7bb57321-f68d-453a-adda-9395daa9d023 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:53.498755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:53.581802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bb57321-f68d-453a-adda-9395daa9d023 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:53.655231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:53.655840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:53.656117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:53.656386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:53.657989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:53.732188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:53.770471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:53.771938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-395d0448-c2c0-4ca5-afe9-e8b66e566270 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:53.790906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 665/1323] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:27:56.011702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422558ab-256c-4401-8f4c-acf3ecf974e0 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:56.012272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422558ab-256c-4401-8f4c-acf3ecf974e0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:56.012272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-422558ab-256c-4401-8f4c-acf3ecf974e0 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:56.013868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 659/1324] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:56.019378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71a22f4-6a82-486f-bdf4-1addeb8a078b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:56.019869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71a22f4-6a82-486f-bdf4-1addeb8a078b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:56.019975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e71a22f4-6a82-486f-bdf4-1addeb8a078b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:56.087354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e71a22f4-6a82-486f-bdf4-1addeb8a078b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:56.088560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e71a22f4-6a82-486f-bdf4-1addeb8a078b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:56.189904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 666/1325] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:27:56.478703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422558ab-256c-4401-8f4c-acf3ecf974e0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:56.480369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-422558ab-256c-4401-8f4c-acf3ecf974e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:56.585722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-422558ab-256c-4401-8f4c-acf3ecf974e0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:56.660887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:56.661499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:56.661779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:56.662051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:56.664448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:56.684518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:56.776856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:56.778226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61d8b529-0e4b-49c3-ac73-37b86e96b5bd service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:56.798921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 660/1326] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:27:59.041273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0cbc01-eab1-4fe9-82dc-457dee297299 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:59.041812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0cbc01-eab1-4fe9-82dc-457dee297299 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:59.041812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0cbc01-eab1-4fe9-82dc-457dee297299 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:59.043131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 667/1327] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:27:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:27:59.048664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37a49d-62e6-4715-8186-baba7b4cfe37 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:59.049105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37a49d-62e6-4715-8186-baba7b4cfe37 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:59.049210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a37a49d-62e6-4715-8186-baba7b4cfe37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:59.115342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4a37a49d-62e6-4715-8186-baba7b4cfe37 None None] MFA Rules not processed for user `e6f8c1e119d74e408654eed7e007c7f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:27:59.116559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4a37a49d-62e6-4715-8186-baba7b4cfe37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:59.181969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:59.214692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 661/1328] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:27:59 2022] POST /identity/v3/auth/tokens => generated 3243 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:27:59.499484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0cbc01-eab1-4fe9-82dc-457dee297299 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:27:59.501296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8d0cbc01-eab1-4fe9-82dc-457dee297299 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:59.536592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:59.620768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0cbc01-eab1-4fe9-82dc-457dee297299 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:27:59.702720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:27:59.703343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:27:59.703617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:27:59.703881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:27:59.705452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:27:59.793115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:27:59.826685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:27:59.828253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e91a65f-969e-46bf-bbd3-65bdf79274d3 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:27:59.847827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 668/1329] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:27:59 2022] GET /identity/v3/auth/tokens => generated 3243 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:02.514721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc77344-2c9c-4f41-9e57-4f56ba91d565 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:02.515344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc77344-2c9c-4f41-9e57-4f56ba91d565 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:02.515344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc77344-2c9c-4f41-9e57-4f56ba91d565 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:02.516656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 662/1330] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:02.521938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bc69c-fa0f-4259-a72f-c9eab9a69f1a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:02.522953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bc69c-fa0f-4259-a72f-c9eab9a69f1a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:02.523653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a53bc69c-fa0f-4259-a72f-c9eab9a69f1a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:02.590927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a53bc69c-fa0f-4259-a72f-c9eab9a69f1a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:02.592154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a53bc69c-fa0f-4259-a72f-c9eab9a69f1a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:02.686753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 669/1331] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:02.968365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc77344-2c9c-4f41-9e57-4f56ba91d565 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:02.969896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ebc77344-2c9c-4f41-9e57-4f56ba91d565 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:03.071545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc77344-2c9c-4f41-9e57-4f56ba91d565 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:03.142345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:03.142968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:03.143245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:03.143515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:03.145076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:03.171237 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:03.253028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:03.254388 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65127750-4f18-4be3-be11-eac47de60e5d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:03.273803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 663/1332] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 306 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:05.529090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df06278c-33dc-4a08-943d-131ef0eab07c None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:05.529647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df06278c-33dc-4a08-943d-131ef0eab07c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:05.529647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df06278c-33dc-4a08-943d-131ef0eab07c None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:05.531021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 670/1333] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:05.536302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dccef2-69bc-4284-81d1-72932b8c6504 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:05.536709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dccef2-69bc-4284-81d1-72932b8c6504 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:05.536843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11dccef2-69bc-4284-81d1-72932b8c6504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:05.602536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-11dccef2-69bc-4284-81d1-72932b8c6504 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:05.603815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-11dccef2-69bc-4284-81d1-72932b8c6504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:05.676670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:05.699671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 664/1334] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:05.985257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df06278c-33dc-4a08-943d-131ef0eab07c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:05.986836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-df06278c-33dc-4a08-943d-131ef0eab07c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:06.026955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:06.090799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df06278c-33dc-4a08-943d-131ef0eab07c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:06.161053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:06.161645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:06.161918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:06.162185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:06.163801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:06.244375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:06.269352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:06.270768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5319807c-f62e-4062-bfdb-5feb2d8890cc service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:06.290137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 671/1335] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:09.108838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f691a7-3c75-4ffe-8aaf-792394083671 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:09.109381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f691a7-3c75-4ffe-8aaf-792394083671 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:09.109381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68f691a7-3c75-4ffe-8aaf-792394083671 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:09.110747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 665/1336] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:09.115985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dae5ef-4858-4530-bbfb-b7dfc79ec8c6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:09.116363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dae5ef-4858-4530-bbfb-b7dfc79ec8c6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:09.116529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1dae5ef-4858-4530-bbfb-b7dfc79ec8c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:09.181138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e1dae5ef-4858-4530-bbfb-b7dfc79ec8c6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:09.182348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e1dae5ef-4858-4530-bbfb-b7dfc79ec8c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:09.279027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 672/1337] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:09 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:09.575395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ffec7e-614a-4b11-83b7-8b652480a9aa None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:09.588288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ffec7e-614a-4b11-83b7-8b652480a9aa None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:09.588288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4ffec7e-614a-4b11-83b7-8b652480a9aa None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:09.588288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 666/1338] 10.222.0.64 () {54 vars in 964 bytes} [Tue Sep 27 16:28:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:09.588560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6df918-253b-464d-b54e-1852631ae01d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:09.588560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6df918-253b-464d-b54e-1852631ae01d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:09.588560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef6df918-253b-464d-b54e-1852631ae01d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:09.642419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:09.667303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ef6df918-253b-464d-b54e-1852631ae01d None None] MFA Rules not processed for user `28f0f9fef2f54e56a7a92d077c25d2b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:09.668521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ef6df918-253b-464d-b54e-1852631ae01d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:09.752533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 673/1339] 10.222.0.64 () {58 vars in 1061 bytes} [Tue Sep 27 16:28:09 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:28:09.761928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ffec7e-614a-4b11-83b7-8b652480a9aa None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:09.765406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b4ffec7e-614a-4b11-83b7-8b652480a9aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:09.871023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4ffec7e-614a-4b11-83b7-8b652480a9aa None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:09.943647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:09.944243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:09.944516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:09.944779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:09.946886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:09.981851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:10.053558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:10.055049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4589ea2-b34f-41c1-8ee7-234d858715c9 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:10.074626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 667/1340] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:09 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:15.259159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea43aff-f8a2-48b0-a47e-c43b51f7860a None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:15.259712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea43aff-f8a2-48b0-a47e-c43b51f7860a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:15.259822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea43aff-f8a2-48b0-a47e-c43b51f7860a None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:15.261249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 674/1341] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:15 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:15.266786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f817ef45-c940-469c-ba66-f97695d770d8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:15.267203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f817ef45-c940-469c-ba66-f97695d770d8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:15.267347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f817ef45-c940-469c-ba66-f97695d770d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:15.334690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f817ef45-c940-469c-ba66-f97695d770d8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:15.335989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f817ef45-c940-469c-ba66-f97695d770d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:15.437667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:15.449792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 668/1342] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 184 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:28:15.736728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea43aff-f8a2-48b0-a47e-c43b51f7860a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:15.738341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-eea43aff-f8a2-48b0-a47e-c43b51f7860a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:15.845249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea43aff-f8a2-48b0-a47e-c43b51f7860a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:15.882854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:15.921046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:15.921864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:15.922150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:15.922667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:15.924355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:16.035545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:16.036884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf2969a3-9f8c-4d80-930d-83db30199ccf service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:16.055186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 675/1343] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:28:19.305806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72136db2-09d3-4ac1-b445-38aa044f13dc None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:19.306356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72136db2-09d3-4ac1-b445-38aa044f13dc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:19.306356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72136db2-09d3-4ac1-b445-38aa044f13dc None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:19.308046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 669/1344] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:19.313279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61590e5-96d1-45a8-ba3f-1dbf9c760e10 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:19.313711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61590e5-96d1-45a8-ba3f-1dbf9c760e10 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:19.313811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f61590e5-96d1-45a8-ba3f-1dbf9c760e10 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:19.381217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f61590e5-96d1-45a8-ba3f-1dbf9c760e10 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:19.383041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f61590e5-96d1-45a8-ba3f-1dbf9c760e10 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:19.401475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:19.479792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 676/1345] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:19 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:19.768354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72136db2-09d3-4ac1-b445-38aa044f13dc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:19.769895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-72136db2-09d3-4ac1-b445-38aa044f13dc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:19.875175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72136db2-09d3-4ac1-b445-38aa044f13dc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:19.950939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:19.951553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:19.951831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:19.952105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:19.953714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:20.021400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:20.094894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:20.096342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb5cec15-7f85-4d15-b0da-0991009ff28d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:20.118093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 670/1346] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:19 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:24.481932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f1cb6-c65c-4204-a2bb-933d5520621e None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:24.482595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f1cb6-c65c-4204-a2bb-933d5520621e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:24.482595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-348f1cb6-c65c-4204-a2bb-933d5520621e None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:24.483921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 677/1347] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:24.489357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd31e35b-dbda-4720-9af8-4ce51b2d0ce5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:24.489794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd31e35b-dbda-4720-9af8-4ce51b2d0ce5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:24.489907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd31e35b-dbda-4720-9af8-4ce51b2d0ce5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:24.556961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-dd31e35b-dbda-4720-9af8-4ce51b2d0ce5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:24.558425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-dd31e35b-dbda-4720-9af8-4ce51b2d0ce5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:24.656041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:24.658424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 671/1348] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:24.947674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f1cb6-c65c-4204-a2bb-933d5520621e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:24.949225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-348f1cb6-c65c-4204-a2bb-933d5520621e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:25.050942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-348f1cb6-c65c-4204-a2bb-933d5520621e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:25.099193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:25.123907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:25.124509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:25.124787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:25.125060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:25.126699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:25.236994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:25.238351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6688a0d-2a40-476b-a336-5dde5a7ff0aa service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:25.257994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 678/1349] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:24 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:30.252762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35519c71-89b8-4543-90e2-d6a4ba62318b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:30.262288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35519c71-89b8-4543-90e2-d6a4ba62318b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:30.262288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35519c71-89b8-4543-90e2-d6a4ba62318b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:30.262288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 672/1350] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:28:30 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:30.263408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8651409a-138b-4d8d-a624-008cc7073722 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:30.263408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8651409a-138b-4d8d-a624-008cc7073722 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:30.263857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8651409a-138b-4d8d-a624-008cc7073722 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:30.376246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8651409a-138b-4d8d-a624-008cc7073722 None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:30.377710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8651409a-138b-4d8d-a624-008cc7073722 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:30.423015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:30.541983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 679/1351] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:28:30 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 280 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:34.684768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0651a813-ed90-4a6e-9515-107954fbb41d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:34.685321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0651a813-ed90-4a6e-9515-107954fbb41d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:34.685321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0651a813-ed90-4a6e-9515-107954fbb41d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:34.686866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 673/1352] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:28:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:34.691786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a432d4ab-7e64-4404-aa11-16f3b2b88a57 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:34.692267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a432d4ab-7e64-4404-aa11-16f3b2b88a57 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:34.692361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a432d4ab-7e64-4404-aa11-16f3b2b88a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:34.776837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a432d4ab-7e64-4404-aa11-16f3b2b88a57 None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:34.778171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a432d4ab-7e64-4404-aa11-16f3b2b88a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:34.854102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:34.892618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 680/1353] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:28:34 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 202 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:36.347814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf36a0b-ddaf-454a-80bc-6366bbc889a3 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:36.348264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf36a0b-ddaf-454a-80bc-6366bbc889a3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:36.348363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faf36a0b-ddaf-454a-80bc-6366bbc889a3 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:36.349723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 674/1354] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:28:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:36.355485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8eefe80-c2c8-45dd-a4a2-add4ad5b1648 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:36.355924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8eefe80-c2c8-45dd-a4a2-add4ad5b1648 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:36.356046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8eefe80-c2c8-45dd-a4a2-add4ad5b1648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:36.432686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f8eefe80-c2c8-45dd-a4a2-add4ad5b1648 None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:36.433891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f8eefe80-c2c8-45dd-a4a2-add4ad5b1648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:36.546767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 681/1355] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:28:36 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:40.026157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe151228-3568-4066-b8b0-873439d0ae0d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:40.027816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe151228-3568-4066-b8b0-873439d0ae0d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:40.028268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe151228-3568-4066-b8b0-873439d0ae0d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:40.029780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 675/1356] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:40.035863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dc45e8-ad40-43cf-85ad-7744784183be None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:40.036766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dc45e8-ad40-43cf-85ad-7744784183be None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:40.037255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20dc45e8-ad40-43cf-85ad-7744784183be None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:40.078758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:40.116910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-20dc45e8-ad40-43cf-85ad-7744784183be None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:40.118611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-20dc45e8-ad40-43cf-85ad-7744784183be None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:40.227020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 682/1357] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:40.527342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646b73-a227-4dfc-9597-4b47caeb1bf9 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:40.527858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646b73-a227-4dfc-9597-4b47caeb1bf9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:40.528887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a646b73-a227-4dfc-9597-4b47caeb1bf9 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:40.530716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 676/1358] 10.222.0.64 () {54 vars in 961 bytes} [Tue Sep 27 16:28:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:40.539789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039db422-22ff-4849-8c37-a76ccd2b08a7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:40.540585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039db422-22ff-4849-8c37-a76ccd2b08a7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:40.541229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-039db422-22ff-4849-8c37-a76ccd2b08a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:40.625727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-039db422-22ff-4849-8c37-a76ccd2b08a7 None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:40.627269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-039db422-22ff-4849-8c37-a76ccd2b08a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:40.678158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:40.753607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 683/1359] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Sep 27 16:28:40 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:40.762022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a646b73-a227-4dfc-9597-4b47caeb1bf9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:40.763964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1a646b73-a227-4dfc-9597-4b47caeb1bf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:40.924948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a646b73-a227-4dfc-9597-4b47caeb1bf9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:41.008754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:41.050557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:41.051992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:41.052902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:41.054126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:41.055762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:41.160630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:41.161325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71c4c717-2430-436b-9bdd-0404923c8611 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:41.184304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 677/1360] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:43.579201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e864b92-d51b-45ae-996e-3ebcb5e947a9 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:43.579714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e864b92-d51b-45ae-996e-3ebcb5e947a9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:43.579714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e864b92-d51b-45ae-996e-3ebcb5e947a9 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:43.581181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 684/1361] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:43.586762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b3a805-3ef5-4b1e-90b5-83323568989d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:43.587248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b3a805-3ef5-4b1e-90b5-83323568989d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:43.587447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b3a805-3ef5-4b1e-90b5-83323568989d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:43.675711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-55b3a805-3ef5-4b1e-90b5-83323568989d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:43.677010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-55b3a805-3ef5-4b1e-90b5-83323568989d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:43.718471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:43.783192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 678/1362] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:44.075909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e864b92-d51b-45ae-996e-3ebcb5e947a9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:44.077573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1e864b92-d51b-45ae-996e-3ebcb5e947a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:44.241884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e864b92-d51b-45ae-996e-3ebcb5e947a9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:44.254349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:44.388943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:44.389545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:44.389817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:44.390079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:44.391660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:44.489065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:44.490588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b84eac-998c-4315-a041-8d7c31f3ce6f service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:44.509975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 685/1363] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 435 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:47.120851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5469e7-c28f-49e1-84e3-188f369adf93 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:47.121406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5469e7-c28f-49e1-84e3-188f369adf93 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:47.121406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5469e7-c28f-49e1-84e3-188f369adf93 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:47.122720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 679/1364] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:47.128081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4880c261-5703-47f5-a03c-5bd6d87ec777 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:47.128526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4880c261-5703-47f5-a03c-5bd6d87ec777 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:47.128612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4880c261-5703-47f5-a03c-5bd6d87ec777 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:47.145872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:47.203484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4880c261-5703-47f5-a03c-5bd6d87ec777 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:47.204937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4880c261-5703-47f5-a03c-5bd6d87ec777 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:47.319439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 686/1365] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:47.610307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b1310-89d2-421c-8a79-520e8d5e9b7d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:47.610723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b1310-89d2-421c-8a79-520e8d5e9b7d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:47.611143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7b1310-89d2-421c-8a79-520e8d5e9b7d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:47.612858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 680/1366] 10.222.0.64 () {54 vars in 961 bytes} [Tue Sep 27 16:28:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:47.619212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df4a489-4991-4171-90bd-4e04cd2de38b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:47.619597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df4a489-4991-4171-90bd-4e04cd2de38b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:47.620384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3df4a489-4991-4171-90bd-4e04cd2de38b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:47.708087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3df4a489-4991-4171-90bd-4e04cd2de38b None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:47.722490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3df4a489-4991-4171-90bd-4e04cd2de38b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:47.766122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:47.855817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 687/1367] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Sep 27 16:28:47 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:47.863503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b1310-89d2-421c-8a79-520e8d5e9b7d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:47.865138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2e7b1310-89d2-421c-8a79-520e8d5e9b7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:48.026900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7b1310-89d2-421c-8a79-520e8d5e9b7d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:48.049474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:48.160002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:48.160637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:48.160798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:48.161068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:48.162717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:48.266292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:48.268042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88eec5c4-2ef6-4247-b267-9bea8f92d626 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:48.288743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 681/1368] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 425 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:48.315683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9782-a572-43db-aad2-aaab79c1dc36 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:48.316184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9782-a572-43db-aad2-aaab79c1dc36 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:48.316286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-685f9782-a572-43db-aad2-aaab79c1dc36 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:48.317722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 688/1369] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:48.323128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea972c9-3aa9-4966-aa38-73683ffe26e7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:48.323409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea972c9-3aa9-4966-aa38-73683ffe26e7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:48.323679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cea972c9-3aa9-4966-aa38-73683ffe26e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:48.359970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:48.394873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-cea972c9-3aa9-4966-aa38-73683ffe26e7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:48.396064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cea972c9-3aa9-4966-aa38-73683ffe26e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:48.514812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 682/1370] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 191 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:48.853516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02860d1-1394-4740-b93e-83c9c3638304 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:48.853516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02860d1-1394-4740-b93e-83c9c3638304 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:48.853516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e02860d1-1394-4740-b93e-83c9c3638304 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:48.856388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 689/1371] 10.222.0.64 () {54 vars in 963 bytes} [Tue Sep 27 16:28:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:48.859603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b31d09-3a90-465b-a56b-78aaf540aee6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:48.859894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b31d09-3a90-465b-a56b-78aaf540aee6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:48.860169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b31d09-3a90-465b-a56b-78aaf540aee6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:48.940904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-43b31d09-3a90-465b-a56b-78aaf540aee6 None None] MFA Rules not processed for user `f8babdc682094c18a4ba710224815a27`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:48.942181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-43b31d09-3a90-465b-a56b-78aaf540aee6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:48.991646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:49.024433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 683/1372] 10.222.0.64 () {58 vars in 1060 bytes} [Tue Sep 27 16:28:48 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:49.035511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02860d1-1394-4740-b93e-83c9c3638304 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:49.038073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e02860d1-1394-4740-b93e-83c9c3638304 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:49.155567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e02860d1-1394-4740-b93e-83c9c3638304 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:49.199803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:49.236687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:49.237922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:49.238203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:49.238474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:49.241881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:49.363906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:49.366496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13de4102-8655-42e3-b046-bf244fbee394 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:49.394947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 690/1373] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:49 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:50.720601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783edf21-51d3-4286-8b52-b27e3c2445c0 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:50.721172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783edf21-51d3-4286-8b52-b27e3c2445c0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:50.721172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783edf21-51d3-4286-8b52-b27e3c2445c0 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:50.722482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 684/1374] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:50.728559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92674791-4859-45c4-9001-d3446e409b86 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:50.729001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92674791-4859-45c4-9001-d3446e409b86 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:50.729094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92674791-4859-45c4-9001-d3446e409b86 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:50.795875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-92674791-4859-45c4-9001-d3446e409b86 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:50.797100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-92674791-4859-45c4-9001-d3446e409b86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:50.818680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:50.898073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 691/1375] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:51.237618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783edf21-51d3-4286-8b52-b27e3c2445c0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:51.237618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-783edf21-51d3-4286-8b52-b27e3c2445c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:51.370885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783edf21-51d3-4286-8b52-b27e3c2445c0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:51.415715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:51.481791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:51.482396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:51.483040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:51.483040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:51.484529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:51.579264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:51.580876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b2ee479-1cf9-4ca6-8a7b-779ce0e22275 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:51.599569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 685/1376] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 401 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:53.967477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d649c946-8803-45d9-819f-5e246e041b87 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:53.968049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d649c946-8803-45d9-819f-5e246e041b87 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:53.968049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d649c946-8803-45d9-819f-5e246e041b87 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:53.969688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 692/1377] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:53.975312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2989f-4b8a-4894-b363-eda2f038cd23 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:53.976098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2989f-4b8a-4894-b363-eda2f038cd23 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:53.976098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb2989f-4b8a-4894-b363-eda2f038cd23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:54.045743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ebb2989f-4b8a-4894-b363-eda2f038cd23 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:54.046984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ebb2989f-4b8a-4894-b363-eda2f038cd23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:54.054458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:54.150978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 686/1378] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:28:54.441703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d649c946-8803-45d9-819f-5e246e041b87 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:54.443437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d649c946-8803-45d9-819f-5e246e041b87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:54.600955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d649c946-8803-45d9-819f-5e246e041b87 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:54.605136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:54.726147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:54.726941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:54.727322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:54.727697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:54.729439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:54.827750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:54.829508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5058970f-dff9-4842-94e3-124d0b4cca97 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:54.848476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 693/1379] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 407 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:28:57.160004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eb6db5-6490-460b-a284-773828afb390 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:57.160512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eb6db5-6490-460b-a284-773828afb390 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:57.160616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75eb6db5-6490-460b-a284-773828afb390 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:57.161961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 687/1380] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:28:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:28:57.167683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf9f42-6dc8-4a60-a413-fd0c625c94b1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:57.168112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf9f42-6dc8-4a60-a413-fd0c625c94b1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:57.168377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96bf9f42-6dc8-4a60-a413-fd0c625c94b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:57.182505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:57.240500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-96bf9f42-6dc8-4a60-a413-fd0c625c94b1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:28:57.241712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-96bf9f42-6dc8-4a60-a413-fd0c625c94b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:57.341099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 694/1381] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:28:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:28:57.637202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eb6db5-6490-460b-a284-773828afb390 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:28:57.638810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-75eb6db5-6490-460b-a284-773828afb390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:57.919540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:57.930386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75eb6db5-6490-460b-a284-773828afb390 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:28:58.049595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:28:58.050204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:28:58.050498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:28:58.050749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:28:58.052375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:28:58.148793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:28:58.150462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b697b328-8847-4934-9453-d89197f1e989 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:28:58.161245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:28:58.172361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 688/1382] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:28:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 536 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:00.875373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9b0fa8-6461-40f0-bb16-fba255caff22 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:00.875918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9b0fa8-6461-40f0-bb16-fba255caff22 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:00.876017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b9b0fa8-6461-40f0-bb16-fba255caff22 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:00.877647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 695/1383] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:00.884217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e4108-cc84-4089-ac9b-c1aec833e257 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:00.884709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e4108-cc84-4089-ac9b-c1aec833e257 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:00.884814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b7e4108-cc84-4089-ac9b-c1aec833e257 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:00.962221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4b7e4108-cc84-4089-ac9b-c1aec833e257 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:00.963505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4b7e4108-cc84-4089-ac9b-c1aec833e257 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:01.073172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 689/1384] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:01.374098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9b0fa8-6461-40f0-bb16-fba255caff22 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:01.375851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0b9b0fa8-6461-40f0-bb16-fba255caff22 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:01.482662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:01.554203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b9b0fa8-6461-40f0-bb16-fba255caff22 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:01.699771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:01.700815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:01.701497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:01.702101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:01.704297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:01.769277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:01.820213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:01.821982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec31dafd-f8e3-4bce-b018-f53ec890f8ff service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:01.842403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 696/1385] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 469 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:04.345639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2133b21-0198-4788-b54b-c7fad12a6b16 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:04.346067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2133b21-0198-4788-b54b-c7fad12a6b16 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:04.346193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2133b21-0198-4788-b54b-c7fad12a6b16 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:04.347562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 690/1386] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:04.353599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4c3dd6-6eb1-411f-86c8-bcd4ee81c1f0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:04.354057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4c3dd6-6eb1-411f-86c8-bcd4ee81c1f0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:04.354157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b4c3dd6-6eb1-411f-86c8-bcd4ee81c1f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:04.423119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5b4c3dd6-6eb1-411f-86c8-bcd4ee81c1f0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:04.424593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5b4c3dd6-6eb1-411f-86c8-bcd4ee81c1f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:04.524876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 697/1387] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:04.819809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2133b21-0198-4788-b54b-c7fad12a6b16 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:04.821414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b2133b21-0198-4788-b54b-c7fad12a6b16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:04.900180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:04.917455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6146733-6cc2-4278-96d2-5c9b9770636b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:04.917733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6146733-6cc2-4278-96d2-5c9b9770636b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:04.917998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6146733-6cc2-4278-96d2-5c9b9770636b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:04.919569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 698/1388] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:29:04.924505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b120d15-0682-4a68-abae-ebc305c9f711 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:04.924787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b120d15-0682-4a68-abae-ebc305c9f711 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:04.925061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b120d15-0682-4a68-abae-ebc305c9f711 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:04.961426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:04.972291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2133b21-0198-4788-b54b-c7fad12a6b16 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:04.993630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6b120d15-0682-4a68-abae-ebc305c9f711 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:04.994869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6b120d15-0682-4a68-abae-ebc305c9f711 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:05.090954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:05.091473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:05.091746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:05.092015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:05.093606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:05.093888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 699/1389] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:29:05.133382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:05.186121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:05.187794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e524599-7fe3-43fa-8679-371060f2fcef service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:05.205677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 691/1390] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:04 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:05.404075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceda6c27-b180-4435-a7ae-424230f85c51 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:05.404579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceda6c27-b180-4435-a7ae-424230f85c51 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:05.404724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceda6c27-b180-4435-a7ae-424230f85c51 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:05.406053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 700/1391] 10.222.0.64 () {54 vars in 962 bytes} [Tue Sep 27 16:29:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:05.412288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465bb72-2e57-4e21-86d7-bdc82c32f7c2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:05.412838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465bb72-2e57-4e21-86d7-bdc82c32f7c2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:05.413121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9465bb72-2e57-4e21-86d7-bdc82c32f7c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:05.498927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9465bb72-2e57-4e21-86d7-bdc82c32f7c2 None None] MFA Rules not processed for user `6a27d7ca876a4320bb05f2afc86bee3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:05.500178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9465bb72-2e57-4e21-86d7-bdc82c32f7c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:05.583271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 692/1392] 10.222.0.64 () {58 vars in 1059 bytes} [Tue Sep 27 16:29:05 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:05.592988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceda6c27-b180-4435-a7ae-424230f85c51 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:05.594429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ceda6c27-b180-4435-a7ae-424230f85c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:05.699466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceda6c27-b180-4435-a7ae-424230f85c51 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:05.707004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:05.775315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:05.775917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:05.776189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:05.776455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:05.778038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:05.890728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:05.892206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a0fb37-440a-42b9-8e9c-e7ea097715a8 service glance] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:05.910513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 701/1393] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 318 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:07.156274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f6998-3ab8-4a8c-8df2-42f75db973d6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:07.156843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f6998-3ab8-4a8c-8df2-42f75db973d6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:07.156843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921f6998-3ab8-4a8c-8df2-42f75db973d6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:07.196828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:07.229021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-921f6998-3ab8-4a8c-8df2-42f75db973d6 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:07.230279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-921f6998-3ab8-4a8c-8df2-42f75db973d6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:07.323472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 693/1394] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:29:07 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:07.372815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe78d11-0982-4d4b-9b45-bc2827c6c094 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:07.373311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe78d11-0982-4d4b-9b45-bc2827c6c094 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:07.373407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abe78d11-0982-4d4b-9b45-bc2827c6c094 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:07.375530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 702/1395] 10.222.0.64 () {54 vars in 943 bytes} [Tue Sep 27 16:29:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:07.382474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8b60a4-55ca-459a-9cb1-d0549cbce490 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:07.382857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8b60a4-55ca-459a-9cb1-d0549cbce490 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:07.383129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a8b60a4-55ca-459a-9cb1-d0549cbce490 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:07.459721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1a8b60a4-55ca-459a-9cb1-d0549cbce490 None None] MFA Rules not processed for user `85bbaf87748242bf830ca84267692c82`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:07.460857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1a8b60a4-55ca-459a-9cb1-d0549cbce490 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:07.522851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:07.574370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 694/1396] 10.222.0.64 () {58 vars in 1040 bytes} [Tue Sep 27 16:29:07 2022] POST /identity/v3/auth/tokens => generated 3304 bytes in 194 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:29:07.583970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe78d11-0982-4d4b-9b45-bc2827c6c094 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:07.585469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-abe78d11-0982-4d4b-9b45-bc2827c6c094 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:07.597417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fff7877-0ca4-45a9-9315-ef633f3534fb None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:07.597692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fff7877-0ca4-45a9-9315-ef633f3534fb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:07.597958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fff7877-0ca4-45a9-9315-ef633f3534fb None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:07.599840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 695/1397] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:29:07.604721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:07.605000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:07.605269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:07.629546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:07.673632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:07.674868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:07.727174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abe78d11-0982-4d4b-9b45-bc2827c6c094 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:07.773326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 696/1398] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:29:07.834351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:07.834984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:07.835261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:07.835530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:07.837207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:07.859607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:07.921019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:07.922627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d478688b-6f68-4b4a-a5f7-ee4b1c9f41b1 service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:07.931451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 703/1399] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:29:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:29:08.060282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:08.061542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:08.093614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:08.173024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c287905-3224-48f4-8542-fa69bb854b3b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:08.289906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:08.290589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:08.290784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:08.291067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:08.292872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:08.330296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:08.403663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:08.405260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99e6c619-876d-42a5-83dd-f06701bdeb91 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:08.424192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 697/1400] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:10.823058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509780f-5da1-40e7-b55a-a4331e344c86 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:10.823636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509780f-5da1-40e7-b55a-a4331e344c86 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:10.823636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e509780f-5da1-40e7-b55a-a4331e344c86 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:10.825070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 704/1401] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:10.830550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f29a8-c136-449d-b602-b4db6b4767ba None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:10.831336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f29a8-c136-449d-b602-b4db6b4767ba None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:10.831610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e44f29a8-c136-449d-b602-b4db6b4767ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:10.908961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e44f29a8-c136-449d-b602-b4db6b4767ba None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:10.910173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e44f29a8-c136-449d-b602-b4db6b4767ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:11.016167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 698/1402] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:11.320451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509780f-5da1-40e7-b55a-a4331e344c86 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:11.323425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e509780f-5da1-40e7-b55a-a4331e344c86 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:11.505977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e509780f-5da1-40e7-b55a-a4331e344c86 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:11.525048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:11.662349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:11.662991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:11.663298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:11.663616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:11.665206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:11.773453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:11.775109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9c6a497-5289-4b74-8483-f8725ea9d35e service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:11.794261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 705/1403] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 474 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:14.498451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ec41c7-5793-4301-b998-dcb43f349081 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:14.499077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ec41c7-5793-4301-b998-dcb43f349081 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:14.499077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92ec41c7-5793-4301-b998-dcb43f349081 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:14.500379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 699/1404] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:14.506744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fef7a1-ed44-44e3-a562-99a0f14e7370 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:14.507293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fef7a1-ed44-44e3-a562-99a0f14e7370 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:14.507293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23fef7a1-ed44-44e3-a562-99a0f14e7370 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:14.534276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:14.578729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-23fef7a1-ed44-44e3-a562-99a0f14e7370 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:14.579988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-23fef7a1-ed44-44e3-a562-99a0f14e7370 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:14.681903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 706/1405] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:14.971861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ec41c7-5793-4301-b998-dcb43f349081 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:14.978019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-92ec41c7-5793-4301-b998-dcb43f349081 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:14.981141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:15.150267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92ec41c7-5793-4301-b998-dcb43f349081 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:15.240970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:15.285915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:15.287222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:15.287573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:15.287919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:15.289649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:15.389163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:15.390804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fc02e2e-a482-4d09-9954-b4e5f0e0afaa service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:15.413425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 700/1406] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:14 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 442 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:17.789659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c769f3a-a284-4ed1-b302-4e3b2925928c None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:17.790189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c769f3a-a284-4ed1-b302-4e3b2925928c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:17.790189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c769f3a-a284-4ed1-b302-4e3b2925928c None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:17.792592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 707/1407] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:29:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:17.797288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf771b-ea9b-411b-8c6c-681fcc5b6179 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:17.797742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf771b-ea9b-411b-8c6c-681fcc5b6179 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:17.797838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8dbf771b-ea9b-411b-8c6c-681fcc5b6179 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:17.872942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8dbf771b-ea9b-411b-8c6c-681fcc5b6179 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:17.874274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8dbf771b-ea9b-411b-8c6c-681fcc5b6179 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:17.916697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:17.983802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 701/1408] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:29:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:18.280467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c769f3a-a284-4ed1-b302-4e3b2925928c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:18.281984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3c769f3a-a284-4ed1-b302-4e3b2925928c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:18.391753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:18.446115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c769f3a-a284-4ed1-b302-4e3b2925928c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:18.582707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:18.584268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:18.584552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:18.584834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:18.586467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:18.658409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:18.690000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:18.691871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9dae3e-4ae4-4133-93fe-acf201d77ba0 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:18.711241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 708/1409] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 431 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:27.982193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=26 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:28.708055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=18 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:32.580524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779d7ac3-e9c6-41d8-b5ee-cb8acddf3314 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:32.581088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779d7ac3-e9c6-41d8-b5ee-cb8acddf3314 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:32.581088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-779d7ac3-e9c6-41d8-b5ee-cb8acddf3314 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:32.582415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 702/1410] 10.222.0.64 () {54 vars in 918 bytes} [Tue Sep 27 16:29:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:32.588430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b5939f-dace-405c-b3bb-1e6b99524598 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:32.588831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b5939f-dace-405c-b3bb-1e6b99524598 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:32.588968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08b5939f-dace-405c-b3bb-1e6b99524598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:32.663050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-08b5939f-dace-405c-b3bb-1e6b99524598 None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:32.664783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-08b5939f-dace-405c-b3bb-1e6b99524598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:32.767806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 709/1411] 10.222.0.64 () {58 vars in 1015 bytes} [Tue Sep 27 16:29:32 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:32.782655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619eb1e6-61ff-4d8e-9940-4a02ca258b64 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:32.782978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619eb1e6-61ff-4d8e-9940-4a02ca258b64 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:32.783253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619eb1e6-61ff-4d8e-9940-4a02ca258b64 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:32.784589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 703/1412] 10.222.0.64 () {54 vars in 918 bytes} [Tue Sep 27 16:29:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:32.789737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f055d-48f7-4059-a077-8949bca1bade None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:32.790015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f055d-48f7-4059-a077-8949bca1bade None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:32.790284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f055d-48f7-4059-a077-8949bca1bade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:32.861197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:32.864102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5d9f055d-48f7-4059-a077-8949bca1bade None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:32.865290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5d9f055d-48f7-4059-a077-8949bca1bade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:32.981379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 710/1413] 10.222.0.64 () {58 vars in 1015 bytes} [Tue Sep 27 16:29:32 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 193 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:35.564512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d685551e-cbf2-4e2f-b5ee-f7a7168435d1 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:35.565083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d685551e-cbf2-4e2f-b5ee-f7a7168435d1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:35.565083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d685551e-cbf2-4e2f-b5ee-f7a7168435d1 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:35.566426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 704/1414] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:29:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:35.571894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8937f66-0455-4590-9337-3db453f17bb6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:35.572307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8937f66-0455-4590-9337-3db453f17bb6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:35.572437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8937f66-0455-4590-9337-3db453f17bb6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:35.637993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d8937f66-0455-4590-9337-3db453f17bb6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:35.639218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d8937f66-0455-4590-9337-3db453f17bb6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:35.694734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:35.737547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 711/1415] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:29:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:36.024647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d685551e-cbf2-4e2f-b5ee-f7a7168435d1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:36.026181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d685551e-cbf2-4e2f-b5ee-f7a7168435d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:36.171846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d685551e-cbf2-4e2f-b5ee-f7a7168435d1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:36.245297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:36.285537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:36.286081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:36.286358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:36.286625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:36.288371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:36.378261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:36.379885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fbd58d3-e6a3-4a74-8403-65dd58327d24 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:36.398237 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 705/1416] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:37.543675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e1544-307c-4746-8bc4-76645fa5caff None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:37.544241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e1544-307c-4746-8bc4-76645fa5caff None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:37.544241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2e1544-307c-4746-8bc4-76645fa5caff None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:37.545584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 712/1417] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:37.551713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea44406-6fd3-436b-8ed5-fccc92d49c70 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:37.552167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea44406-6fd3-436b-8ed5-fccc92d49c70 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:37.552265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ea44406-6fd3-436b-8ed5-fccc92d49c70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:37.628768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5ea44406-6fd3-436b-8ed5-fccc92d49c70 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:37.629978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5ea44406-6fd3-436b-8ed5-fccc92d49c70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:37.657106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:37.727383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 706/1418] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:37.885833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e1544-307c-4746-8bc4-76645fa5caff None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:37.887297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0e2e1544-307c-4746-8bc4-76645fa5caff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:38.034157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2e1544-307c-4746-8bc4-76645fa5caff None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:38.075849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:38.147329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:38.147891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:38.148168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:38.148438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:38.150022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:38.241113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:38.243399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff2fa9dc-c92e-4326-87d8-9bccf7781138 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:38.261006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 713/1419] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 376 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:39.396090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911b9f-9f40-445c-b7a3-445d1ec13290 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:39.396494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911b9f-9f40-445c-b7a3-445d1ec13290 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:39.396494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1911b9f-9f40-445c-b7a3-445d1ec13290 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:39.398366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 707/1420] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:39.404801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ca0615-6d90-4e7c-8ac6-433b62c84273 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:39.405242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ca0615-6d90-4e7c-8ac6-433b62c84273 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:39.405357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7ca0615-6d90-4e7c-8ac6-433b62c84273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:39.458741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:39.481140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f7ca0615-6d90-4e7c-8ac6-433b62c84273 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:39.482299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f7ca0615-6d90-4e7c-8ac6-433b62c84273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:39.582370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 714/1421] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:39.737364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1911b9f-9f40-445c-b7a3-445d1ec13290 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:39.738926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e1911b9f-9f40-445c-b7a3-445d1ec13290 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:39.884108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1911b9f-9f40-445c-b7a3-445d1ec13290 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:39.896103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:39.996421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:39.997027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:39.997305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:39.997575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:39.999212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:40.087280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:40.088864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef555f18-fb56-46fe-b403-de99559b4523 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:40.106302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 708/1422] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:41.218701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64e2ef2-b641-4657-9215-f7221b6f71e3 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:41.219305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64e2ef2-b641-4657-9215-f7221b6f71e3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:41.219305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e64e2ef2-b641-4657-9215-f7221b6f71e3 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:41.220696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 715/1423] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:41.227291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897503f-5cff-41d3-b31e-e590d34403ea None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:41.227759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897503f-5cff-41d3-b31e-e590d34403ea None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:41.227857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1897503f-5cff-41d3-b31e-e590d34403ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:41.247934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:41.304517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1897503f-5cff-41d3-b31e-e590d34403ea None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:41.305976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1897503f-5cff-41d3-b31e-e590d34403ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:41.403262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 709/1424] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:41.556105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64e2ef2-b641-4657-9215-f7221b6f71e3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:41.557529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e64e2ef2-b641-4657-9215-f7221b6f71e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:41.665952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:41.700584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e64e2ef2-b641-4657-9215-f7221b6f71e3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:41.811790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:41.812310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:41.812585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:41.812849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:41.814445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:41.888665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:41.902351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:41.903961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37cadb57-bcbf-4f40-b8e0-d21bb0f2209f service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:41.922545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 716/1425] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:43.030852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7889c43-fb16-4277-bf2b-091e7db8658e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:43.031457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7889c43-fb16-4277-bf2b-091e7db8658e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:43.031457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7889c43-fb16-4277-bf2b-091e7db8658e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:43.032725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 710/1426] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:43.038973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca23f691-45d9-4c60-a756-18f4af4454cd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:43.039423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca23f691-45d9-4c60-a756-18f4af4454cd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:43.039519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca23f691-45d9-4c60-a756-18f4af4454cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:43.113900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ca23f691-45d9-4c60-a756-18f4af4454cd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:43.115121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ca23f691-45d9-4c60-a756-18f4af4454cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:43.212843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 717/1427] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:43.365696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7889c43-fb16-4277-bf2b-091e7db8658e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:43.367919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a7889c43-fb16-4277-bf2b-091e7db8658e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:43.452461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:43.514079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7889c43-fb16-4277-bf2b-091e7db8658e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:43.624507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:43.625114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:43.625396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:43.625665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:43.627280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:43.675494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:43.720228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:43.721808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a135cd3a-62d9-4573-850f-822a92e04a4d service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:43.739346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 711/1428] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:44.848265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef642104-981d-410a-b920-ab25a0d38781 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:44.848849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef642104-981d-410a-b920-ab25a0d38781 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:44.848849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef642104-981d-410a-b920-ab25a0d38781 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:44.850812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 718/1429] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:44.857249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf58ccd-ac0f-4741-8ead-38f213d2d531 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:44.857697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf58ccd-ac0f-4741-8ead-38f213d2d531 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:44.857813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bf58ccd-ac0f-4741-8ead-38f213d2d531 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:44.932332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7bf58ccd-ac0f-4741-8ead-38f213d2d531 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:44.933527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7bf58ccd-ac0f-4741-8ead-38f213d2d531 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:45.028768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 712/1430] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:45.179911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef642104-981d-410a-b920-ab25a0d38781 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:45.181409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ef642104-981d-410a-b920-ab25a0d38781 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:45.229583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:45.324068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef642104-981d-410a-b920-ab25a0d38781 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:45.435453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:45.435993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:45.436267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:45.436533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:45.438123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:45.445114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:45.526087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:45.527722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82f134b5-7c6b-4e91-97d2-941cc3227900 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:45.545722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 719/1431] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:45 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 367 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:46.657621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4bbaf8-dccc-462f-be0d-d355466ec653 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:46.658179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4bbaf8-dccc-462f-be0d-d355466ec653 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:46.658179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee4bbaf8-dccc-462f-be0d-d355466ec653 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:46.659519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 713/1432] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:46.665743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace22a7-f995-4761-a297-5574877d1754 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:46.666292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace22a7-f995-4761-a297-5574877d1754 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:46.666292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ace22a7-f995-4761-a297-5574877d1754 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:46.740912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1ace22a7-f995-4761-a297-5574877d1754 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:46.742060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1ace22a7-f995-4761-a297-5574877d1754 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:46.808643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:46.840198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 720/1433] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:46.988603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4bbaf8-dccc-462f-be0d-d355466ec653 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:46.990055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ee4bbaf8-dccc-462f-be0d-d355466ec653 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:47.008622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:47.136647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee4bbaf8-dccc-462f-be0d-d355466ec653 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:47.222800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:47.248407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:47.248949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:47.249222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:47.249491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:47.251106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:47.339850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:47.341476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7d0d24a-84a3-4299-84ef-55585ee5f7ba service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:47.359550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 714/1434] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:46 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:48.468139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d63c0f-6de2-430a-998a-8df8bb7145d1 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:48.468699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d63c0f-6de2-430a-998a-8df8bb7145d1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:48.468699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99d63c0f-6de2-430a-998a-8df8bb7145d1 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:48.470021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 721/1435] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:48.476687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bba614-da3c-4f72-bbd1-9f5b80feaa51 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:48.477212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bba614-da3c-4f72-bbd1-9f5b80feaa51 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:48.477319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47bba614-da3c-4f72-bbd1-9f5b80feaa51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:48.553306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-47bba614-da3c-4f72-bbd1-9f5b80feaa51 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:48.555782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-47bba614-da3c-4f72-bbd1-9f5b80feaa51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:48.597452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:48.653794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 715/1436] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:48.803584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d63c0f-6de2-430a-998a-8df8bb7145d1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:48.806079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-99d63c0f-6de2-430a-998a-8df8bb7145d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:48.948965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99d63c0f-6de2-430a-998a-8df8bb7145d1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:49.002500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:49.062596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:49.063233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:49.063508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:49.063774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:49.065374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:49.152957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:49.154536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f45d245-5270-45e9-bd3d-2b7c71b57427 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:49.172553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 722/1437] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 369 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:50.289814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e440ce-cbc7-4fb5-8b34-e28a18ad3631 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:50.290402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e440ce-cbc7-4fb5-8b34-e28a18ad3631 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:50.290402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e440ce-cbc7-4fb5-8b34-e28a18ad3631 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:50.291761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 716/1438] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:29:50 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:50.297982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0206a1-a0d1-46ef-b190-5e4bf2e2fb79 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:50.298419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0206a1-a0d1-46ef-b190-5e4bf2e2fb79 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:50.298537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0206a1-a0d1-46ef-b190-5e4bf2e2fb79 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:50.380955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7d0206a1-a0d1-46ef-b190-5e4bf2e2fb79 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:50.382227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7d0206a1-a0d1-46ef-b190-5e4bf2e2fb79 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:50.384271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:50.476172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 723/1439] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:29:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:50.624553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e440ce-cbc7-4fb5-8b34-e28a18ad3631 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:50.626118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-24e440ce-cbc7-4fb5-8b34-e28a18ad3631 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:50.771867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e440ce-cbc7-4fb5-8b34-e28a18ad3631 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:50.799293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:50.886080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:50.886629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:50.886931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:50.887249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:50.888827 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:50.978362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:50.979967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b358a00e-040c-4ca0-8327-9572cf4dd690 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:50.997871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 717/1440] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:50 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 374 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:29:53.073828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f2c309-a7c8-4b75-be18-884eed55d327 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:53.074466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f2c309-a7c8-4b75-be18-884eed55d327 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:53.074466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03f2c309-a7c8-4b75-be18-884eed55d327 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:53.075738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 724/1441] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:29:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:53.081080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf0e98d-f881-45ce-983e-a5ee2e849c02 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:53.081515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf0e98d-f881-45ce-983e-a5ee2e849c02 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:53.081620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbf0e98d-f881-45ce-983e-a5ee2e849c02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:53.124961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:53.150299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fbf0e98d-f881-45ce-983e-a5ee2e849c02 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:53.151711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fbf0e98d-f881-45ce-983e-a5ee2e849c02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:53.248806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 718/1442] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:29:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:53.546970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f2c309-a7c8-4b75-be18-884eed55d327 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:53.548560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-03f2c309-a7c8-4b75-be18-884eed55d327 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:53.651686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03f2c309-a7c8-4b75-be18-884eed55d327 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:53.683948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:53.724457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:53.725060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:53.725369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:53.726146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:53.727765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:53.835349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:53.837097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05f840a7-04e6-4229-bf84-534f4238e952 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:53.855254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 725/1443] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:56.165136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7115dbe-1be1-4efd-b264-bf11f5b3fa2e None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:56.165767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7115dbe-1be1-4efd-b264-bf11f5b3fa2e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:56.165767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7115dbe-1be1-4efd-b264-bf11f5b3fa2e None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:56.167092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 719/1444] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:29:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:56.172288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761e869-f716-45d3-a301-54c3573a43ba None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:56.172679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761e869-f716-45d3-a301-54c3573a43ba None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:56.172859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9761e869-f716-45d3-a301-54c3573a43ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:56.239082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9761e869-f716-45d3-a301-54c3573a43ba None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:56.240292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9761e869-f716-45d3-a301-54c3573a43ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:56.252858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:56.335788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 726/1445] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:29:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:56.616643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7115dbe-1be1-4efd-b264-bf11f5b3fa2e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:56.618219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d7115dbe-1be1-4efd-b264-bf11f5b3fa2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:56.721384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7115dbe-1be1-4efd-b264-bf11f5b3fa2e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:29:56.734677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:56.794932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:29:56.795533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:56.795811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:56.796074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:56.797657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:56.906186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:29:56.907568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f77354-df62-43b5-aa1d-34f304607661 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:29:56.925879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 720/1446] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:56 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:29:59.438989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f537b84-073a-41bb-ae4b-1e6987421e3a None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:59.439491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f537b84-073a-41bb-ae4b-1e6987421e3a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:59.439592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f537b84-073a-41bb-ae4b-1e6987421e3a None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:59.440914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 727/1447] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:29:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:29:59.446748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f52c47-dec3-4c6f-9bd5-5e19cef881f1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:29:59.447212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f52c47-dec3-4c6f-9bd5-5e19cef881f1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:29:59.447322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f52c47-dec3-4c6f-9bd5-5e19cef881f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:29:59.494507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:29:59.513219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-51f52c47-dec3-4c6f-9bd5-5e19cef881f1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:29:59.514416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-51f52c47-dec3-4c6f-9bd5-5e19cef881f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:29:59.609350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 721/1448] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:29:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:29:59.908933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f537b84-073a-41bb-ae4b-1e6987421e3a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:29:59.910467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3f537b84-073a-41bb-ae4b-1e6987421e3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:00.013695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f537b84-073a-41bb-ae4b-1e6987421e3a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:00.054111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:00.087451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:00.088047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:00.088321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:00.088588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:00.090163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:00.196874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:00.198256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8eba2455-366d-42d9-84dd-d19e9d540409 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:00.217167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 728/1449] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:29:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:30:02.581744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ad597-fa87-4e55-93aa-386e596077da None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:02.582308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ad597-fa87-4e55-93aa-386e596077da None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:02.582308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21ad597-fa87-4e55-93aa-386e596077da None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:02.583665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 722/1450] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:02.589647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c12157-7866-4dfe-86c4-2f69840c77c6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:02.590084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c12157-7866-4dfe-86c4-2f69840c77c6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:02.590190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9c12157-7866-4dfe-86c4-2f69840c77c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:02.656218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-e9c12157-7866-4dfe-86c4-2f69840c77c6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:02.657417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e9c12157-7866-4dfe-86c4-2f69840c77c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:02.677408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:02.872086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 729/1451] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 283 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:03.156566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ad597-fa87-4e55-93aa-386e596077da None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:03.158119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d21ad597-fa87-4e55-93aa-386e596077da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:03.263350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21ad597-fa87-4e55-93aa-386e596077da None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:03.284314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:03.339728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:03.340343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:03.340624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:03.340894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:03.342501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:03.453644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:03.455112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27efa03a-22f4-4a69-948c-649ea02fe64d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:03.472647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 723/1452] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:05.758557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49ea354-1e0b-4ab8-a79a-3ed96d9772bd None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:05.759169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49ea354-1e0b-4ab8-a79a-3ed96d9772bd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:05.759169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b49ea354-1e0b-4ab8-a79a-3ed96d9772bd None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:05.760475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 730/1453] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:05 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:05.765839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b779e8-9028-468e-b6bd-7073b30264bc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:05.766266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b779e8-9028-468e-b6bd-7073b30264bc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:05.766405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55b779e8-9028-468e-b6bd-7073b30264bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:05.833823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:05.836866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-55b779e8-9028-468e-b6bd-7073b30264bc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:05.838081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-55b779e8-9028-468e-b6bd-7073b30264bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:05.933783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 724/1454] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:06.219474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49ea354-1e0b-4ab8-a79a-3ed96d9772bd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:06.221020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b49ea354-1e0b-4ab8-a79a-3ed96d9772bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:06.323844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b49ea354-1e0b-4ab8-a79a-3ed96d9772bd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:06.373855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:06.398339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:06.398994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:06.399271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:06.399544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:06.401172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:06.509260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:06.510605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6b857-c77f-42c6-846d-48695316a669 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:06.528867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 731/1455] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:06 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:08.774046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558aa6e-0407-4c9d-91ed-38a87fb1e59a None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:08.774596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558aa6e-0407-4c9d-91ed-38a87fb1e59a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:08.774596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d558aa6e-0407-4c9d-91ed-38a87fb1e59a None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:08.775993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 725/1456] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:08.781246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b29c37-374c-4baa-981a-a5d8ff9b4051 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:08.781676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b29c37-374c-4baa-981a-a5d8ff9b4051 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:08.781812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9b29c37-374c-4baa-981a-a5d8ff9b4051 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:08.847007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c9b29c37-374c-4baa-981a-a5d8ff9b4051 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:08.848227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c9b29c37-374c-4baa-981a-a5d8ff9b4051 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:08.877951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:08.942834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 732/1457] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:09.229487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558aa6e-0407-4c9d-91ed-38a87fb1e59a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:09.231129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d558aa6e-0407-4c9d-91ed-38a87fb1e59a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:09.337851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d558aa6e-0407-4c9d-91ed-38a87fb1e59a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:09.367754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:09.413948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:09.414568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:09.414876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:09.415148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:09.416742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:09.526978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:09.528335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b886a70c-d152-46b4-a2d0-b614a7f1117a service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:09.546075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 726/1458] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:11.804218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbde4544-c1f7-492b-9b31-ea73d542a4d1 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:11.804810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbde4544-c1f7-492b-9b31-ea73d542a4d1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:11.804810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbde4544-c1f7-492b-9b31-ea73d542a4d1 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:11.806097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 733/1459] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:11.811522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cda38f-3f08-455f-9f3c-92de8987b9ec None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:11.811970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cda38f-3f08-455f-9f3c-92de8987b9ec None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:11.812087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02cda38f-3f08-455f-9f3c-92de8987b9ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:11.880219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-02cda38f-3f08-455f-9f3c-92de8987b9ec None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:11.881548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-02cda38f-3f08-455f-9f3c-92de8987b9ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:11.889690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:11.977817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 727/1460] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:12.264465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbde4544-c1f7-492b-9b31-ea73d542a4d1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:12.266121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fbde4544-c1f7-492b-9b31-ea73d542a4d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:12.368909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbde4544-c1f7-492b-9b31-ea73d542a4d1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:12.422719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:12.440619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:12.441216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:12.441488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:12.441843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:12.443641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:12.550238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:12.551612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c1a6e54-c876-4ba1-818a-782a59be54f4 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:12.569064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 734/1461] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:12 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:14.877886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c45fdc-f9dc-4361-87ca-a767eebc3418 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:14.878471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c45fdc-f9dc-4361-87ca-a767eebc3418 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:14.878471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c45fdc-f9dc-4361-87ca-a767eebc3418 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:14.879847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 728/1462] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:14.885566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4d0c55-84f5-4e7b-a965-8c0332d24847 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:14.886016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4d0c55-84f5-4e7b-a965-8c0332d24847 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:14.886135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4d0c55-84f5-4e7b-a965-8c0332d24847 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:14.952591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-dd4d0c55-84f5-4e7b-a965-8c0332d24847 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:14.953794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dd4d0c55-84f5-4e7b-a965-8c0332d24847 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:14.992746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:15.051469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 735/1463] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:15.337158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c45fdc-f9dc-4361-87ca-a767eebc3418 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:15.338760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-64c45fdc-f9dc-4361-87ca-a767eebc3418 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:15.455494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c45fdc-f9dc-4361-87ca-a767eebc3418 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:15.493189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:15.528719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:15.529323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:15.529596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:15.529858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:15.531477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:15.639532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:15.640967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7952330-0efa-4f16-b281-5131cb4e6ee4 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:15.659413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 729/1464] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:17.847627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2931bf8-9cad-4619-b73b-08b88b982064 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:17.848200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2931bf8-9cad-4619-b73b-08b88b982064 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:17.848200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2931bf8-9cad-4619-b73b-08b88b982064 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:17.850235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 736/1465] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:17.855587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f06b77d-ce22-4178-b8ac-68c774bc1485 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:17.856029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f06b77d-ce22-4178-b8ac-68c774bc1485 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:17.856182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f06b77d-ce22-4178-b8ac-68c774bc1485 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:17.922239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4f06b77d-ce22-4178-b8ac-68c774bc1485 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:17.923495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f06b77d-ce22-4178-b8ac-68c774bc1485 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:17.942395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:18.022509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 730/1466] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:18.300798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2931bf8-9cad-4619-b73b-08b88b982064 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:18.302908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b2931bf8-9cad-4619-b73b-08b88b982064 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:18.406878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2931bf8-9cad-4619-b73b-08b88b982064 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:18.473175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:18.479272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:18.479869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:18.480146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:18.480419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:18.482062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:18.591122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:18.592476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136eb77b-9c3c-4b67-bb50-9f99dd260dc2 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:18.610525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 737/1467] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:18 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:20.779863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d104574-691d-4887-b9ea-02a8bb5ea53d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:20.780415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d104574-691d-4887-b9ea-02a8bb5ea53d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:20.780415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d104574-691d-4887-b9ea-02a8bb5ea53d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:20.781741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 731/1468] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:30:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:20.787662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d762f-463e-4f87-9b1b-0cfcca91e40b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:20.788094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d762f-463e-4f87-9b1b-0cfcca91e40b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:20.788241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c13d762f-463e-4f87-9b1b-0cfcca91e40b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:20.858531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c13d762f-463e-4f87-9b1b-0cfcca91e40b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:20.859785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c13d762f-463e-4f87-9b1b-0cfcca91e40b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:20.908149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:20.958046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 738/1469] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:30:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:30:21.246904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d104574-691d-4887-b9ea-02a8bb5ea53d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:21.248449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d104574-691d-4887-b9ea-02a8bb5ea53d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:21.356477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d104574-691d-4887-b9ea-02a8bb5ea53d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:21.407732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:21.436588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:21.437401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:21.437833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:21.438455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:21.440134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:21.551748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:21.553538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4782ce1-e053-43ec-8a3a-0673e90d16cb service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:21.571753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 732/1470] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:30.956489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=22 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:31.569556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=66 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:41.932436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a32e1-9a07-49d3-8e8a-90f1e3451231 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:41.932984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a32e1-9a07-49d3-8e8a-90f1e3451231 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:41.932984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96a32e1-9a07-49d3-8e8a-90f1e3451231 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:41.934326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 739/1471] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:30:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:41.940039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84493162-93f1-440a-a919-b6f5c54ff0f2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:41.940504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84493162-93f1-440a-a919-b6f5c54ff0f2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:41.940612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84493162-93f1-440a-a919-b6f5c54ff0f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:42.010615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-84493162-93f1-440a-a919-b6f5c54ff0f2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:42.011853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-84493162-93f1-440a-a919-b6f5c54ff0f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:42.117352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 733/1472] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:30:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 178 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:42.432554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e6980-4cde-4864-a369-b1bf8e8991c4 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:42.432834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e6980-4cde-4864-a369-b1bf8e8991c4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:42.433108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-548e6980-4cde-4864-a369-b1bf8e8991c4 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:42.434452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 740/1473] 10.222.0.64 () {54 vars in 962 bytes} [Tue Sep 27 16:30:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:42.439501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6befbbc-e8a2-4faf-acbf-0b35e627da25 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:42.439791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6befbbc-e8a2-4faf-acbf-0b35e627da25 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:42.440066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6befbbc-e8a2-4faf-acbf-0b35e627da25 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:42.526762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d6befbbc-e8a2-4faf-acbf-0b35e627da25 None None] MFA Rules not processed for user `6a27d7ca876a4320bb05f2afc86bee3d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:42.527978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d6befbbc-e8a2-4faf-acbf-0b35e627da25 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:42.549265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:42.607796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 734/1474] 10.222.0.64 () {58 vars in 1059 bytes} [Tue Sep 27 16:30:42 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:42.615554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e6980-4cde-4864-a369-b1bf8e8991c4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:42.617056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-548e6980-4cde-4864-a369-b1bf8e8991c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:42.728031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-548e6980-4cde-4864-a369-b1bf8e8991c4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:42.805600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:42.806203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:42.806471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:42.806731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:42.808445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:42.864541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:42.922012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:42.923487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c07f980-4936-49be-b7fb-53ae87c6a7d1 service glance] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:42.943150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 741/1475] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 328 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:45.200423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c69820-9c20-4370-9772-6627a47978ba None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:45.200970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c69820-9c20-4370-9772-6627a47978ba None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:45.200970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c69820-9c20-4370-9772-6627a47978ba None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:45.202303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 735/1476] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:30:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:45.208045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bbe782-1559-472c-86cc-2b70e9dde0aa None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:45.208506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bbe782-1559-472c-86cc-2b70e9dde0aa None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:45.208827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bbe782-1559-472c-86cc-2b70e9dde0aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:45.274800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a7bbe782-1559-472c-86cc-2b70e9dde0aa None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:45.276007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a7bbe782-1559-472c-86cc-2b70e9dde0aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:45.375403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 742/1477] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:30:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:45.648218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c69820-9c20-4370-9772-6627a47978ba None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:45.649746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-15c69820-9c20-4370-9772-6627a47978ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:45.775780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c69820-9c20-4370-9772-6627a47978ba None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:45.828266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:45.876484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:45.877091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:45.877370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:45.877636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:45.878331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:45.880160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:30:45.880160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:30:45.880160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:30:45.881313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72f77646-c71e-407c-a669-ba8f8b9ecc65 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:45.900635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 736/1478] 10.222.0.64 () {56 vars in 1108 bytes} [Tue Sep 27 16:30:45 2022] GET /identity/v3/projects => generated 2553 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:30:48.064725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5810b79c-2bb8-4d82-bcb0-d54fadba37c5 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:48.065280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5810b79c-2bb8-4d82-bcb0-d54fadba37c5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:48.065280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5810b79c-2bb8-4d82-bcb0-d54fadba37c5 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:48.067198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 743/1479] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:30:48 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:48.073027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0e2e0d-cb79-44e2-83cc-846b5029ab5b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:48.073431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0e2e0d-cb79-44e2-83cc-846b5029ab5b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:48.073570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f0e2e0d-cb79-44e2-83cc-846b5029ab5b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:48.139336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0f0e2e0d-cb79-44e2-83cc-846b5029ab5b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:48.140533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0f0e2e0d-cb79-44e2-83cc-846b5029ab5b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:48.237720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 737/1480] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:30:48 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:30:48.535118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5810b79c-2bb8-4d82-bcb0-d54fadba37c5 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:48.536618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5810b79c-2bb8-4d82-bcb0-d54fadba37c5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:48.546185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:48.683360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5810b79c-2bb8-4d82-bcb0-d54fadba37c5 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:48.760301 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:48.791585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:48.792179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:48.792453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:48.792717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:48.794293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:48.881791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:48.883384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-556cea48-ea06-4fdf-b1e3-3ff74fbeade2 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:48.900792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 744/1481] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:51.222048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7782b357-c202-4709-b26a-ec43ad8edb12 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:51.222621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7782b357-c202-4709-b26a-ec43ad8edb12 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:51.222621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7782b357-c202-4709-b26a-ec43ad8edb12 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:51.224607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 738/1482] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:30:51 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:51.229805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4e5670-4e95-4c1a-a894-ce8fdd1cf5eb None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:51.230198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4e5670-4e95-4c1a-a894-ce8fdd1cf5eb None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:51.230345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4e5670-4e95-4c1a-a894-ce8fdd1cf5eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:51.294472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ca4e5670-4e95-4c1a-a894-ce8fdd1cf5eb None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:51.295754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ca4e5670-4e95-4c1a-a894-ce8fdd1cf5eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:51.342480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:51.392626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 745/1483] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:30:51 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:51.684551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7782b357-c202-4709-b26a-ec43ad8edb12 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:51.686436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7782b357-c202-4709-b26a-ec43ad8edb12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:51.710962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:51.792135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7782b357-c202-4709-b26a-ec43ad8edb12 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:51.872907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:51.873519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:51.873795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:51.874066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:51.875694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:51.949628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:51.984454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:51.985894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30df9b2e-be9d-44ee-b89f-6ab175e6748d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:52.004631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 739/1484] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:54.208477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc31cb9-6f7f-40fc-8a34-5f27ad97a9ad None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:54.209136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc31cb9-6f7f-40fc-8a34-5f27ad97a9ad None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:54.209136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bc31cb9-6f7f-40fc-8a34-5f27ad97a9ad None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:54.210371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 746/1485] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:30:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:54.215787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5837f1-2b4d-4bf8-9372-6a32b9f15d3e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:54.216215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5837f1-2b4d-4bf8-9372-6a32b9f15d3e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:54.216323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b5837f1-2b4d-4bf8-9372-6a32b9f15d3e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:54.281899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2b5837f1-2b4d-4bf8-9372-6a32b9f15d3e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:54.283131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2b5837f1-2b4d-4bf8-9372-6a32b9f15d3e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:54.382409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 740/1486] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:30:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 168 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:54.677940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc31cb9-6f7f-40fc-8a34-5f27ad97a9ad None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:54.679519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9bc31cb9-6f7f-40fc-8a34-5f27ad97a9ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:54.782874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bc31cb9-6f7f-40fc-8a34-5f27ad97a9ad None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:54.854562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:54.856064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:54.856667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:54.856944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:54.857216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:54.858844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:54.966812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:54.968175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3c00b29-7979-4c30-bd02-811daf2b2ec0 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:54.986419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 747/1487] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:30:57.344738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41233e78-32cd-423f-b27e-504db7a942f5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:57.345169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41233e78-32cd-423f-b27e-504db7a942f5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:57.345289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41233e78-32cd-423f-b27e-504db7a942f5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:57.346627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 741/1488] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:30:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:30:57.352666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ad9cf-e95c-4153-8a44-7553ee91b3a8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:57.353118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ad9cf-e95c-4153-8a44-7553ee91b3a8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:57.353231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a33ad9cf-e95c-4153-8a44-7553ee91b3a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:57.418385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a33ad9cf-e95c-4153-8a44-7553ee91b3a8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:30:57.419621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a33ad9cf-e95c-4153-8a44-7553ee91b3a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:57.474072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:57.518056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 748/1489] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:30:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:30:57.811684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41233e78-32cd-423f-b27e-504db7a942f5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:30:57.813216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-41233e78-32cd-423f-b27e-504db7a942f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:57.850102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:57.917487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41233e78-32cd-423f-b27e-504db7a942f5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:30:57.991650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:30:57.992252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:30:57.992526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:30:57.992789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:30:57.994629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:30:58.074204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:30:58.103534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:30:58.104962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b7e5def-0a06-4278-9136-8d6f7e18f192 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:30:58.125962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 742/1490] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:30:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:31:00.645898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6af8b4-c09b-427e-9992-1378772bf703 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:31:00.646474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6af8b4-c09b-427e-9992-1378772bf703 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:31:00.646474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d6af8b4-c09b-427e-9992-1378772bf703 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:31:00.647938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 749/1491] 10.222.0.64 () {54 vars in 927 bytes} [Tue Sep 27 16:31:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:31:00.653363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e171377-6ec5-4e60-bea1-ac71c72e6783 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:31:00.658124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e171377-6ec5-4e60-bea1-ac71c72e6783 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:31:00.658124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e171377-6ec5-4e60-bea1-ac71c72e6783 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:31:00.720267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6e171377-6ec5-4e60-bea1-ac71c72e6783 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:31:00.721469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6e171377-6ec5-4e60-bea1-ac71c72e6783 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:31:00.819116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 743/1492] 10.222.0.64 () {58 vars in 1024 bytes} [Tue Sep 27 16:31:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:31:01.108265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6af8b4-c09b-427e-9992-1378772bf703 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:31:01.109789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9d6af8b4-c09b-427e-9992-1378772bf703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:31:01.214268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d6af8b4-c09b-427e-9992-1378772bf703 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:31:01.286651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:31:01.287281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:31:01.287559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:31:01.287830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:31:01.289448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:31:01.297906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:31:01.399362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:31:01.400750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cea9665-af0f-4d55-8eb2-6cb6a0c5136e service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:31:01.419274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 750/1493] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:31:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:31:10.817613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=82 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:31:11.416978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=50 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:32:52.594819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:52.595379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:52.595379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:52.679095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:32:52.680305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:52.783204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 744/1494] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:32:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:32:52.791754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:52.792252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:52.792360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:52.812688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:32:52.813880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:52.823229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 751/1495] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:32:52 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:32:52.827760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:52.837510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9c9a3e9-cc11-4ac6-955c-f571c549a2a5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:52.842082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:52.842669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:52.842976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:52.843244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:52.843876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:32:52.844173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:52.845223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:52.845223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:32:52.845223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:32:52.846042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:52.858167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 745/1496] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:32:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:32:52.866258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:52.875795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37ab398e-595f-4e2b-832a-c91d62a6b210 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:52.880370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:52.880952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:52.881227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:52.881491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:52.882703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:52.884448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:52.884448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:32:52.884448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:32:52.885641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:52.985565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 752/1497] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:32:52 2022] POST /identity/v3/projects => generated 377 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:32:52.989738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:53.000562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d064bfb0-0295-4a35-b804-f9da873a5776 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:53.049125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:32:53.056550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:32:53.062562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:32:53.063158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:53.157868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 746/1498] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:32:52 2022] POST /identity/v3/users => generated 289 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:32:53.161893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:53.172784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bbe1d7-db08-4855-a97e-fcde83b973b2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:53.177408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:53.178000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.178278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.178545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.179861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:53.180912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:53.180912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:32:53.180912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:32:53.181802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:53.194934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 753/1499] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:32:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:32:53.200139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:53.208740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b654bea-e047-4fe4-a961-dd21a9ae80a5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:53.213566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:53.214158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.214435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.214707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.215373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:53.216377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:53.216377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:32:53.216377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:32:53.217294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6017d7a8-6705-48e1-93e2-c7ad8d3a86a2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:53.231050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 747/1500] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:32:53 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:32:53.234797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:53.245279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76b12b28-6c13-45ce-97be-b916f5b91eaf admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:53.249835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:53.250469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.250744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.251043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] PATH_INFO: `/v3/projects/f829782908fd439bad22962ad4193cf1/users/b8c2a092cd0e40aa89bcceb92fcfa291/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.265149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f829782908fd439bad22962ad4193cf1, user_id=b8c2a092cd0e40aa89bcceb92fcfa291, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:53.266160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:53.266160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:32:53.266160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:32:53.268200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6dc578c7-7620-4dfc-86b4-ca234ccce66d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:53.339221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 754/1501] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:32:53 2022] PUT /identity/v3/projects/f829782908fd439bad22962ad4193cf1/users/b8c2a092cd0e40aa89bcceb92fcfa291/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:32:53.352071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87d841e-336a-4999-b0b7-9ebc22613b5e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.352380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87d841e-336a-4999-b0b7-9ebc22613b5e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.352680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87d841e-336a-4999-b0b7-9ebc22613b5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.379856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a87d841e-336a-4999-b0b7-9ebc22613b5e None None] MFA Rules not processed for user `b8c2a092cd0e40aa89bcceb92fcfa291`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:32:53.381092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a87d841e-336a-4999-b0b7-9ebc22613b5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:53.445618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 748/1502] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:32:53 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:32:53.458577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f51bea3-a936-451c-9432-a5f08b63e83d None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.458884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f51bea3-a936-451c-9432-a5f08b63e83d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.459156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f51bea3-a936-451c-9432-a5f08b63e83d None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.461039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 755/1503] 10.222.0.64 () {54 vars in 963 bytes} [Tue Sep 27 16:32:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:32:53.466054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c539557-25c6-4c12-942f-26ac31cf96cb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.466366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c539557-25c6-4c12-942f-26ac31cf96cb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.466641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c539557-25c6-4c12-942f-26ac31cf96cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.525333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1c539557-25c6-4c12-942f-26ac31cf96cb None None] MFA Rules not processed for user `f8babdc682094c18a4ba710224815a27`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:32:53.526527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1c539557-25c6-4c12-942f-26ac31cf96cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:53.602087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 749/1504] 10.222.0.64 () {58 vars in 1060 bytes} [Tue Sep 27 16:32:53 2022] POST /identity/v3/auth/tokens => generated 3120 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:32:53.609867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f51bea3-a936-451c-9432-a5f08b63e83d None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:53.620358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f51bea3-a936-451c-9432-a5f08b63e83d None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:53.623869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:53.624448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.624718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.624982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.632971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:53.634372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:53.642866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 756/1505] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:32:53 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 33 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:32:53.687890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e9bf7-50ce-412c-9481-4e497a3017b5 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.688173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e9bf7-50ce-412c-9481-4e497a3017b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.688445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-699e9bf7-50ce-412c-9481-4e497a3017b5 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.689687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 750/1506] 10.222.0.64 () {52 vars in 830 bytes} [Tue Sep 27 16:32:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:32:53.693290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:53.755289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d44699-1a91-4ea9-b67b-e19e14c57d2c service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:53.760464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:53.761085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:53.761364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:53.761637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] PATH_INFO: `/v3/users/b8c2a092cd0e40aa89bcceb92fcfa291` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:53.762756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8c2a092cd0e40aa89bcceb92fcfa291)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:53.763789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:53.763789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:32:53.763789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:32:53.765006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:53.951720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8c2a092cd0e40aa89bcceb92fcfa291'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:53.952357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8c2a092cd0e40aa89bcceb92fcfa291'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:53.966974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8c2a092cd0e40aa89bcceb92fcfa291'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:53.967728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8c2a092cd0e40aa89bcceb92fcfa291'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.032442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8c2a092cd0e40aa89bcceb92fcfa291'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.055141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=3,keystone:DELETE=9 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:32:54.063126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 757/1507] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:32:53 2022] DELETE /identity/v3/users/b8c2a092cd0e40aa89bcceb92fcfa291 => generated 0 bytes in 370 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:32:54.067708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:54.067991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:54.068267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:54.087598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:32:54.088793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:54.097644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:32:54.146063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 751/1508] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:32:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:32:54.162072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:54.163769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:54.243181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27aedf26-0f15-4d3b-9b60-6b8f933159d4 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:54.246816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d05c9ae-d0af-4820-8a73-0d9cbb4a2b90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:54.247467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d05c9ae-d0af-4820-8a73-0d9cbb4a2b90 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:54.247745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d05c9ae-d0af-4820-8a73-0d9cbb4a2b90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:54.248014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d05c9ae-d0af-4820-8a73-0d9cbb4a2b90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:54.257653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d05c9ae-d0af-4820-8a73-0d9cbb4a2b90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:54.259077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d05c9ae-d0af-4820-8a73-0d9cbb4a2b90 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:54.268463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 758/1509] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:32:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:32:54.751919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:32:54.753411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:32:54.766451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4f4386c-1fa6-4214-a25d-dc18e29be2a6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:32:54.771585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:32:54.772201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:32:54.772475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:32:54.772782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] PATH_INFO: `/v3/projects/f829782908fd439bad22962ad4193cf1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:32:54.774192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f829782908fd439bad22962ad4193cf1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:32:54.775214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:32:54.775214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:32:54.775214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:32:54.776426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:32:54.894110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f829782908fd439bad22962ad4193cf1', 'request_id': 'req-416daf14-ebcc-4047-876a-006fadb7cac1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.901238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f829782908fd439bad22962ad4193cf1', 'request_id': 'req-416daf14-ebcc-4047-876a-006fadb7cac1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.966486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f829782908fd439bad22962ad4193cf1', 'request_id': 'req-416daf14-ebcc-4047-876a-006fadb7cac1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.986430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] The token cache is being invalidate because project f829782908fd439bad22962ad4193cf1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:32:54.987728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.987728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-416daf14-ebcc-4047-876a-006fadb7cac1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:32:54.989090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 752/1510] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:32:54 2022] DELETE /identity/v3/projects/f829782908fd439bad22962ad4193cf1 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:33:04.264179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=1,keystone:SELECT=37 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:33:04.983224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=47,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:34:25.393760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0294accb-d494-4792-970e-69aa7cef95b0 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:25.394292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0294accb-d494-4792-970e-69aa7cef95b0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:25.394292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0294accb-d494-4792-970e-69aa7cef95b0 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:25.395690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 759/1511] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:34:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:25.401209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c7208-54c0-4a43-8f2d-2696808c6f0e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:25.401674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c7208-54c0-4a43-8f2d-2696808c6f0e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:25.401779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-223c7208-54c0-4a43-8f2d-2696808c6f0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:25.421417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-223c7208-54c0-4a43-8f2d-2696808c6f0e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:25.422608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-223c7208-54c0-4a43-8f2d-2696808c6f0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:25.479680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 753/1512] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:34:25 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:25.637642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0294accb-d494-4792-970e-69aa7cef95b0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:25.639354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0294accb-d494-4792-970e-69aa7cef95b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:25.744027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0294accb-d494-4792-970e-69aa7cef95b0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:25.749702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8825db1c-e9f2-469f-bdef-1468ceb17f4f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:25.750306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8825db1c-e9f2-469f-bdef-1468ceb17f4f service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:25.750582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8825db1c-e9f2-469f-bdef-1468ceb17f4f service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:25.750879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8825db1c-e9f2-469f-bdef-1468ceb17f4f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:25.759824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8825db1c-e9f2-469f-bdef-1468ceb17f4f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:25.761458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8825db1c-e9f2-469f-bdef-1468ceb17f4f service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:25.770680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 760/1513] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:25 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:34:26.883176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150b88c-7a03-49cc-83ba-1cc2bf7cf1e2 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:26.883745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150b88c-7a03-49cc-83ba-1cc2bf7cf1e2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:26.883745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b150b88c-7a03-49cc-83ba-1cc2bf7cf1e2 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:26.885102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 754/1514] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:34:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:26.892611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74ae1a5-ecd8-48aa-a4de-cd46b7ae885d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:26.893081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74ae1a5-ecd8-48aa-a4de-cd46b7ae885d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:26.893192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74ae1a5-ecd8-48aa-a4de-cd46b7ae885d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:26.913865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a74ae1a5-ecd8-48aa-a4de-cd46b7ae885d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:26.915010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a74ae1a5-ecd8-48aa-a4de-cd46b7ae885d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:26.923301 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 761/1515] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:34:26 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:34:27.077725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150b88c-7a03-49cc-83ba-1cc2bf7cf1e2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:27.092255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b150b88c-7a03-49cc-83ba-1cc2bf7cf1e2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:27.097769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4469907b-249a-4c94-be72-f45ae3272832 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:27.098402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4469907b-249a-4c94-be72-f45ae3272832 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:27.098699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4469907b-249a-4c94-be72-f45ae3272832 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:27.099003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4469907b-249a-4c94-be72-f45ae3272832 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:27.108100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4469907b-249a-4c94-be72-f45ae3272832 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:27.109797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4469907b-249a-4c94-be72-f45ae3272832 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:27.120202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 755/1516] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:27 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:34:27.147359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728b1120-3535-40fc-868b-12b8422eef56 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:27.147636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728b1120-3535-40fc-868b-12b8422eef56 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:27.147903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728b1120-3535-40fc-868b-12b8422eef56 None None] PATH_INFO: `/v3` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:27.149257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 762/1517] 10.222.0.33 () {54 vars in 925 bytes} [Tue Sep 27 16:34:27 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:34:27.181119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6315b46-e80f-4c35-86f4-fbca63abc36e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:27.181480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6315b46-e80f-4c35-86f4-fbca63abc36e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:27.181767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6315b46-e80f-4c35-86f4-fbca63abc36e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:27.238101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b6315b46-e80f-4c35-86f4-fbca63abc36e None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:27.239369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b6315b46-e80f-4c35-86f4-fbca63abc36e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:27.312866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 756/1518] 10.222.0.33 () {58 vars in 1015 bytes} [Tue Sep 27 16:34:27 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 133 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:31.043728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623ab9b3-b3fb-427d-b7bd-62ebc27a6d44 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:31.044246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623ab9b3-b3fb-427d-b7bd-62ebc27a6d44 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:31.044351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623ab9b3-b3fb-427d-b7bd-62ebc27a6d44 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:31.045610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 763/1519] 10.222.0.64 () {54 vars in 961 bytes} [Tue Sep 27 16:34:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:31.050939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bd3a6a-4c9f-4be0-ab44-9f1ca5d36917 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:31.051400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bd3a6a-4c9f-4be0-ab44-9f1ca5d36917 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:31.051520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7bd3a6a-4c9f-4be0-ab44-9f1ca5d36917 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:31.070806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f7bd3a6a-4c9f-4be0-ab44-9f1ca5d36917 None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:31.071977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f7bd3a6a-4c9f-4be0-ab44-9f1ca5d36917 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:31.080350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 757/1520] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Sep 27 16:34:31 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:31.088515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623ab9b3-b3fb-427d-b7bd-62ebc27a6d44 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:31.102638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623ab9b3-b3fb-427d-b7bd-62ebc27a6d44 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:31.107697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e03e1d1-c251-4db6-b049-d0c008fc0b38 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:31.111488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e03e1d1-c251-4db6-b049-d0c008fc0b38 service placement] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:31.111488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e03e1d1-c251-4db6-b049-d0c008fc0b38 service placement] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:31.111488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e03e1d1-c251-4db6-b049-d0c008fc0b38 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:31.117202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e03e1d1-c251-4db6-b049-d0c008fc0b38 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:31.118767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e03e1d1-c251-4db6-b049-d0c008fc0b38 service placement] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:31.128153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 764/1521] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:34:38.612006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa7a29-5e3d-4a2f-bb70-e89da7c1c368 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:38.612557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa7a29-5e3d-4a2f-bb70-e89da7c1c368 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:38.612557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa7a29-5e3d-4a2f-bb70-e89da7c1c368 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:38.613935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 758/1522] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:34:38 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:38.620479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098798d-ea25-4af1-accc-ac9cdba182c2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:38.620961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098798d-ea25-4af1-accc-ac9cdba182c2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:38.621067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b098798d-ea25-4af1-accc-ac9cdba182c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:38.640485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b098798d-ea25-4af1-accc-ac9cdba182c2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:38.641671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b098798d-ea25-4af1-accc-ac9cdba182c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:38.650048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 765/1523] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:34:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:38.804753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa7a29-5e3d-4a2f-bb70-e89da7c1c368 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:38.819525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa7a29-5e3d-4a2f-bb70-e89da7c1c368 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:38.824961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08655ebf-9055-4092-9531-0fa17a64e06d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:38.825554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08655ebf-9055-4092-9531-0fa17a64e06d service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:38.825831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08655ebf-9055-4092-9531-0fa17a64e06d service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:38.826099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08655ebf-9055-4092-9531-0fa17a64e06d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:38.834174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08655ebf-9055-4092-9531-0fa17a64e06d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:38.836445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08655ebf-9055-4092-9531-0fa17a64e06d service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:38.846267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 759/1524] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:34:39.927437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd2ed2b-a977-423e-a18f-ade1eab2b75b None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:39.928010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd2ed2b-a977-423e-a18f-ade1eab2b75b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:39.928010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdd2ed2b-a977-423e-a18f-ade1eab2b75b None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:39.929405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 766/1525] 127.0.0.1 () {54 vars in 901 bytes} [Tue Sep 27 16:34:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:39.935581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f8143-4792-46fc-8372-59e01ffe5ee8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:39.936035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f8143-4792-46fc-8372-59e01ffe5ee8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:39.936135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-088f8143-4792-46fc-8372-59e01ffe5ee8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:39.955961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-088f8143-4792-46fc-8372-59e01ffe5ee8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:39.957077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-088f8143-4792-46fc-8372-59e01ffe5ee8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:39.967368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 760/1526] 127.0.0.1 () {58 vars in 998 bytes} [Tue Sep 27 16:34:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:40.119633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd2ed2b-a977-423e-a18f-ade1eab2b75b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:40.133952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdd2ed2b-a977-423e-a18f-ade1eab2b75b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:40.139277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bd0d772-b19f-409d-8e1b-27105e40de0d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:40.139861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd0d772-b19f-409d-8e1b-27105e40de0d service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:40.140134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd0d772-b19f-409d-8e1b-27105e40de0d service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:40.140402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bd0d772-b19f-409d-8e1b-27105e40de0d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:40.148794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd0d772-b19f-409d-8e1b-27105e40de0d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:40.150353 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bd0d772-b19f-409d-8e1b-27105e40de0d service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:40.159514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 767/1527] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:34:44.241084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bdfc43-d6e7-414f-9df3-2c1b9a67daa4 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:44.241654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bdfc43-d6e7-414f-9df3-2c1b9a67daa4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:44.241654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45bdfc43-d6e7-414f-9df3-2c1b9a67daa4 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:44.243058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 761/1528] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:34:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:44.248318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d4d380-9793-4e43-b8c4-11c00cff1e23 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:44.248759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d4d380-9793-4e43-b8c4-11c00cff1e23 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:44.248877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d4d380-9793-4e43-b8c4-11c00cff1e23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:44.267727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-53d4d380-9793-4e43-b8c4-11c00cff1e23 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:44.268927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-53d4d380-9793-4e43-b8c4-11c00cff1e23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:44.277130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 768/1529] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:34:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:34:44.563082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bdfc43-d6e7-414f-9df3-2c1b9a67daa4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:44.564753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-45bdfc43-d6e7-414f-9df3-2c1b9a67daa4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:44.593597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:34:44.622002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45bdfc43-d6e7-414f-9df3-2c1b9a67daa4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:44.625578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e97de5-19e4-475c-9c30-9e66d984578d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:44.626176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e97de5-19e4-475c-9c30-9e66d984578d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:44.626449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e97de5-19e4-475c-9c30-9e66d984578d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:44.626717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e97de5-19e4-475c-9c30-9e66d984578d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:44.636498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e97de5-19e4-475c-9c30-9e66d984578d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:44.637920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e97de5-19e4-475c-9c30-9e66d984578d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:44.647882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 762/1530] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:34:46.807937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ab16af-ba88-4e9a-9940-63c6ff1718ff None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:46.808502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ab16af-ba88-4e9a-9940-63c6ff1718ff None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:46.808502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1ab16af-ba88-4e9a-9940-63c6ff1718ff None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:46.809812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 769/1531] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:34:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:46.815104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73001460-2b09-4ee7-8fdc-b683b9842dc4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:46.815534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73001460-2b09-4ee7-8fdc-b683b9842dc4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:46.815640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73001460-2b09-4ee7-8fdc-b683b9842dc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:46.835408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-73001460-2b09-4ee7-8fdc-b683b9842dc4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:46.836588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-73001460-2b09-4ee7-8fdc-b683b9842dc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:46.844312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 763/1532] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:34:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:47.137487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ab16af-ba88-4e9a-9940-63c6ff1718ff None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:47.148074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1ab16af-ba88-4e9a-9940-63c6ff1718ff None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:47.151559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d53f600-1eae-46de-8ce4-0b6d32389201 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:47.152146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d53f600-1eae-46de-8ce4-0b6d32389201 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:47.152424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d53f600-1eae-46de-8ce4-0b6d32389201 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:47.173385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d53f600-1eae-46de-8ce4-0b6d32389201 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:47.173385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d53f600-1eae-46de-8ce4-0b6d32389201 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:47.173385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d53f600-1eae-46de-8ce4-0b6d32389201 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:47.173385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 770/1533] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:34:50.808141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866426ac-586d-4f54-98cd-076d83b97d5f None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:50.808821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866426ac-586d-4f54-98cd-076d83b97d5f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:50.808821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-866426ac-586d-4f54-98cd-076d83b97d5f None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:50.810013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 764/1534] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:34:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:50.815259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce40c2f-67c5-442d-8a16-7d0f56c9188a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:50.815660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce40c2f-67c5-442d-8a16-7d0f56c9188a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:50.815811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fce40c2f-67c5-442d-8a16-7d0f56c9188a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:50.834940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fce40c2f-67c5-442d-8a16-7d0f56c9188a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:50.836138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fce40c2f-67c5-442d-8a16-7d0f56c9188a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:50.843869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 771/1535] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:34:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:51.151928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866426ac-586d-4f54-98cd-076d83b97d5f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:51.153581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-866426ac-586d-4f54-98cd-076d83b97d5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:51.210216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-866426ac-586d-4f54-98cd-076d83b97d5f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:51.213747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19b5f86-30ce-48a9-8857-0cd7c8a69e5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:51.214344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b5f86-30ce-48a9-8857-0cd7c8a69e5c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:51.214617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b5f86-30ce-48a9-8857-0cd7c8a69e5c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:51.214913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19b5f86-30ce-48a9-8857-0cd7c8a69e5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:51.223607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b5f86-30ce-48a9-8857-0cd7c8a69e5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:51.225203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a19b5f86-30ce-48a9-8857-0cd7c8a69e5c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:51.234367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 765/1536] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:51 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:34:53.386450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04317e82-5df7-44c5-ab0e-d4e9821e6035 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:53.387013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04317e82-5df7-44c5-ab0e-d4e9821e6035 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:53.387123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04317e82-5df7-44c5-ab0e-d4e9821e6035 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:53.388404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 772/1537] 127.0.0.1 () {54 vars in 917 bytes} [Tue Sep 27 16:34:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:34:53.393750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fce0b9-fe97-4c71-9013-a2269f0721ba None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:53.394204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fce0b9-fe97-4c71-9013-a2269f0721ba None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:53.394314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fce0b9-fe97-4c71-9013-a2269f0721ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:53.414902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a6fce0b9-fe97-4c71-9013-a2269f0721ba None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:34:53.416094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a6fce0b9-fe97-4c71-9013-a2269f0721ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:53.424104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 766/1538] 127.0.0.1 () {58 vars in 1014 bytes} [Tue Sep 27 16:34:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:34:53.736742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04317e82-5df7-44c5-ab0e-d4e9821e6035 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:34:53.738438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-04317e82-5df7-44c5-ab0e-d4e9821e6035 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:34:53.752046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04317e82-5df7-44c5-ab0e-d4e9821e6035 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:34:53.755586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ca42b32-32f3-47ff-a786-ae1c4e1187c0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:34:53.756161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca42b32-32f3-47ff-a786-ae1c4e1187c0 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:34:53.756443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca42b32-32f3-47ff-a786-ae1c4e1187c0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:34:53.756700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ca42b32-32f3-47ff-a786-ae1c4e1187c0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:34:53.765350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca42b32-32f3-47ff-a786-ae1c4e1187c0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:34:53.766798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ca42b32-32f3-47ff-a786-ae1c4e1187c0 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:34:53.775937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 773/1539] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:34:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:35:03.410065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=47 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:35:03.771963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=82 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:19.185520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dd9874-39d4-4a77-b24d-61d82af99280 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.186187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dd9874-39d4-4a77-b24d-61d82af99280 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.186187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57dd9874-39d4-4a77-b24d-61d82af99280 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.216538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-57dd9874-39d4-4a77-b24d-61d82af99280 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.217675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-57dd9874-39d4-4a77-b24d-61d82af99280 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.232135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 767/1540] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.250693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.251812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.252303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.266373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f81e38-301e-429f-8ac6-b4717a7d9f0f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.266643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f81e38-301e-429f-8ac6-b4717a7d9f0f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.266890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6f81e38-301e-429f-8ac6-b4717a7d9f0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.285071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.287810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.298280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d6f81e38-301e-429f-8ac6-b4717a7d9f0f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.299599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d6f81e38-301e-429f-8ac6-b4717a7d9f0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.304520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 774/1541] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.312129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.313133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 768/1542] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:19.326872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03828edb-3bda-418a-b02e-3d80dcb59bbf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.326872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03828edb-3bda-418a-b02e-3d80dcb59bbf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.326872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03828edb-3bda-418a-b02e-3d80dcb59bbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.333600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b0028cc-7df5-4e41-ac3c-0e140c32e3fc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.349375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.350430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.351338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.353177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.354274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:19.354827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.356092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:36:19.356092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:36:19.356092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:36:19.357396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.368705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-03828edb-3bda-418a-b02e-3d80dcb59bbf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.370073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03828edb-3bda-418a-b02e-3d80dcb59bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.380532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 769/1543] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 59 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.381897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.382180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.382448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.386597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 775/1544] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 75 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:36:19.387815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.402236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c2d4ae7-3430-4483-8706-3ac2ee0eceb8 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.416748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.417306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.418417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.418736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.419267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.419738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.420588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:19.421238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.428320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f14f6a2-ad0f-4133-879f-0b8320a9bcdc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.441882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 770/1545] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.441882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.451502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 776/1546] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:19.459099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.459730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.460224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.461983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56b0e923-bf1e-43b6-995a-b85bb62d88c6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.468384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.468384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.468384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.468384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.468927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.474798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:36:19.474798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:36:19.474798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:36:19.474798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c5df7cd-f00e-472c-9e37-37f189c8214b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.507821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.510719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.529840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 777/1547] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:19.531570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.546492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15652be4-3538-4b38-a1c9-94f103b45ad0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.552799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c2bd47-6f4b-4b70-8716-cdc32f666f89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.553566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2bd47-6f4b-4b70-8716-cdc32f666f89 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.553848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2bd47-6f4b-4b70-8716-cdc32f666f89 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.554118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c2bd47-6f4b-4b70-8716-cdc32f666f89 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.554761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c2bd47-6f4b-4b70-8716-cdc32f666f89 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.557868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c2bd47-6f4b-4b70-8716-cdc32f666f89 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.606324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 771/1548] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/projects => generated 357 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:19.607927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.608324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.608662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.631104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.632348 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.655318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 778/1549] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/projects => generated 387 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:19.657164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cdf20-2520-4285-910f-155297b85fb8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.657447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cdf20-2520-4285-910f-155297b85fb8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.657716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b8cdf20-2520-4285-910f-155297b85fb8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.679197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9b8cdf20-2520-4285-910f-155297b85fb8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.682715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9b8cdf20-2520-4285-910f-155297b85fb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.721389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 772/1550] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.722685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.734373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce7ff82b-f072-4467-8fdd-03dd5ac188b2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.753933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 779/1551] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.755263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01186a-3715-4696-9e8b-5855e614a813 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.755882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01186a-3715-4696-9e8b-5855e614a813 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.755882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf01186a-3715-4696-9e8b-5855e614a813 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.782721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cf01186a-3715-4696-9e8b-5855e614a813 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.783976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cf01186a-3715-4696-9e8b-5855e614a813 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.794628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 780/1552] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.796510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.808515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.808515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.813140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.813975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.814253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.814525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.815723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:19.816031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.817675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2403d899-2bdb-43fa-93ea-dda8fff4fa8a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.823332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.824847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.830815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 773/1553] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 109 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:19.832082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.832366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.833271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 781/1554] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.833401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.834009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.844026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6fe8d3-35da-45fa-8c5f-6897c93544a7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.852489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.853786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.854129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.854466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.855786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.856791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:36:19.856791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:36:19.856791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:36:19.858011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e2ad12-2a56-4269-bd0b-30087c38e43b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.861174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:19.862415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:19.871509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 774/1555] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:19.872293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:19.881718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f62f5c2-4f47-4dbd-a074-eed33440077f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:19.886529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11506979-7858-4ab9-9a2e-a66dfa7fbc5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:19.887171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11506979-7858-4ab9-9a2e-a66dfa7fbc5e admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.887447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11506979-7858-4ab9-9a2e-a66dfa7fbc5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.887719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11506979-7858-4ab9-9a2e-a66dfa7fbc5e admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.888344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11506979-7858-4ab9-9a2e-a66dfa7fbc5e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:19.890298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11506979-7858-4ab9-9a2e-a66dfa7fbc5e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:19.989795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 775/1556] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/users => generated 294 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:19.989994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 782/1557] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/users => generated 279 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:19.990132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.990264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d67e71-7b57-4b4d-89a2-8626b08ea1bc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:19.990371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.990479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d67e71-7b57-4b4d-89a2-8626b08ea1bc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:19.990585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:19.990683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8d67e71-7b57-4b4d-89a2-8626b08ea1bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.009344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d8d67e71-7b57-4b4d-89a2-8626b08ea1bc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:20.014277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:20.014277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:20.014488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d8d67e71-7b57-4b4d-89a2-8626b08ea1bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:20.018490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 783/1558] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:20.022508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 776/1559] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:20.022508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.022697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c4e795-87bf-4c27-b9b4-97c8d8ab529f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.022697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c4e795-87bf-4c27-b9b4-97c8d8ab529f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.022697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8c4e795-87bf-4c27-b9b4-97c8d8ab529f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.032060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3758fe0-2881-40bd-9e7f-e42605e0fb1e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.036738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.037363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.037641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.037913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.038804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.041323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.041448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c8c4e795-87bf-4c27-b9b4-97c8d8ab529f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:20.042540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c8c4e795-87bf-4c27-b9b4-97c8d8ab529f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:20.050459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 784/1560] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:20.051751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3856b115-6a55-469d-ac1e-86398e6a0977 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.052120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3856b115-6a55-469d-ac1e-86398e6a0977 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.052740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3856b115-6a55-469d-ac1e-86398e6a0977 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.056444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:20.072058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3856b115-6a55-469d-ac1e-86398e6a0977 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:20.073241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3856b115-6a55-469d-ac1e-86398e6a0977 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:20.081150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 785/1561] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:20.082397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.082688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.082988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.102304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:20.104248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:20.111928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 786/1562] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:20.112687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.121956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7899a76e-a84c-4cb7-b5fd-3950165a9872 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.126570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.127186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.127462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.127729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.128354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.129938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.139738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 787/1563] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 27 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:20.140510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.150273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-071605a8-c856-4843-bde3-f4f6c034822d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.155169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.155772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.156087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.156356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.156974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.158535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.170805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 788/1564] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:20.170805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.181262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 777/1565] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/projects => generated 377 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:20.181262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.183843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff21f9b-6fd0-49b0-bf2f-cd3a3d1b5b0f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.200697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:20.240563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.241149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.241424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.241691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.242371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:20.242669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.244310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.245895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea54476-92e4-43c2-9f63-4b2e5e018746 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.250893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.251500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.251892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.252169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.252838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:20.253229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.254947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.255806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 789/1566] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:20.256627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.266361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0ca17e-5937-4454-9232-cdc8f654aa4c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.273784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 778/1567] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:20.273784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.279304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.279916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.280035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c20fb6-800e-4d0e-9004-5a10dacf2697 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.280193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.280460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.281103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:20.281403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.283168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.284563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.285158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.285435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.286029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.286664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:20.286996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.288820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.297776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 790/1568] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:20.298629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.308377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de75e9de-7203-413a-b3d1-d9be4734eb8b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.311135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 779/1569] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:20.313481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.314182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.314797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.315088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.315373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.316002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:20.316306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.317880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.325900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21328b9f-1a95-4bd6-9206-6a6e46f96fda admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.330557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 791/1570] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:20.331520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.337808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.338629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.338986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.339254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.339897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.341522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.348047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d967da8a-6491-4aa6-ac66-5a14babca4d0 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.352235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 780/1571] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:20.353745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.354446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.355081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.355365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.355634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.356396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.357984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.364218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c29a5aa-edb9-4068-86c4-3177db004ec6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.369071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.370911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 792/1572] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:20 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:20.371058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.371058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.371058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.371275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.371349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.373008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.382067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a79a18-4d5a-432e-9d11-9aab87bf1343 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.388057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.388642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.388918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.389190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.389815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.391847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.455136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 781/1573] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/users => generated 289 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:20.456033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.466257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d3cc600-f7f0-476f-9662-d641940e359e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.470795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.476148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.476148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.476148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.476148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.476148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.641503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 793/1574] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/projects => generated 383 bytes in 270 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:20.641503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.658045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 782/1575] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/projects => generated 387 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:20.658045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.704580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb5479a4-b2af-40bc-8d45-90e01b7f9183 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.722297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a118f466-f756-4044-9452-a0c807b298e5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.726866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.727946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.728233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.728501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.729261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.731337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.764514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.768654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.768654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.768654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.768654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.768654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.773783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:20.816533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 783/1576] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/projects => generated 405 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:20.817610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.829241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=6,keystone:SELECT=94 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:20.880699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee23c8db-9f1d-4f9d-99da-4b2a06106bae admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.885434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.886032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.886310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.886581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.887280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.889188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:20.898873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 794/1577] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/projects => generated 385 bytes in 255 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:20.898873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:20.959113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b62f62e2-78a7-4c6f-b31f-a9b1a4ac554b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:20.964419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:20.965068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:20.965342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:20.965611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] PATH_INFO: `/v3/projects/a1ae034405f34841bd96ab9c70711049/users/8c3fb5ebdd5746a6b3bd667e5338c42e/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:20.980086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1ae034405f34841bd96ab9c70711049, user_id=8c3fb5ebdd5746a6b3bd667e5338c42e, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:20.982733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.002166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 784/1578] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:20 2022] POST /identity/v3/projects => generated 385 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.002166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.059847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46088036-7419-474a-b2c4-5e8cd5a09b22 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.066436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 795/1579] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:20 2022] PUT /identity/v3/projects/a1ae034405f34841bd96ab9c70711049/users/8c3fb5ebdd5746a6b3bd667e5338c42e/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:21.077086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.128154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.128961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.129240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.129513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] PATH_INFO: `/v3/projects/44114768a46c440ebe3222da40801a9f/users/6a153563a2c040de84179a4a2b456030/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.129626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eab787a6-9f74-4f12-a41b-e7d4c67f928c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.135562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.136159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.136443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.136711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.137339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.138961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.143947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] RBAC: Authorizing `identity:create_grant(project_id=44114768a46c440ebe3222da40801a9f, user_id=6a153563a2c040de84179a4a2b456030, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.145018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:36:21.145018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:36:21.145018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:36:21.147014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.149017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 796/1580] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.150370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.159652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40222d4-85fb-4884-a7a5-f36aae129e53 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.164781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.170378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.170378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.170378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.170378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.170378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.203990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 785/1581] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:20 2022] PUT /identity/v3/projects/44114768a46c440ebe3222da40801a9f/users/6a153563a2c040de84179a4a2b456030/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:21.205169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.234190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:21.238133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 797/1582] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/users => generated 292 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.239292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.248817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23819b69-1a4e-472a-8f4e-c1b1ece0865e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.269225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6966cd8d-db3a-4f6f-8696-13a001f7ef14 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.275023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.275889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.276171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.276583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.277217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.279200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.295651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:21.307674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.308268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.308553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.308825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.309458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.311546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.357760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 786/1583] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/users => generated 294 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.359925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.369275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61dd2a1-b0a2-4b15-abc8-e53b8a96461b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.378351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef38ffee-d5d8-4295-bbbb-8b9ce79ca752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.379082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef38ffee-d5d8-4295-bbbb-8b9ce79ca752 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.379370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef38ffee-d5d8-4295-bbbb-8b9ce79ca752 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.379646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef38ffee-d5d8-4295-bbbb-8b9ce79ca752 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.380276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef38ffee-d5d8-4295-bbbb-8b9ce79ca752 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.381505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 798/1584] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/users => generated 303 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.387042 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.387145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef38ffee-d5d8-4295-bbbb-8b9ce79ca752 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.392693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e354a1a8-dfbf-4fc6-8de5-df7b97044911 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.397314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.397892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.398166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.398432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.399145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.401969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.477332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 787/1585] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/users => generated 293 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.478941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.479228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.479503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.506401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 799/1586] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/users => generated 293 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.507489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.512822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] MFA Rules not processed for user `8c3fb5ebdd5746a6b3bd667e5338c42e`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:21.514066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:21.521288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb1981cd-32a4-4f41-9b61-09eca00722c6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.527909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-145fd280-876b-46a1-9411-338a19ceaeb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.528509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145fd280-876b-46a1-9411-338a19ceaeb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.528784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145fd280-876b-46a1-9411-338a19ceaeb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.529051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-145fd280-876b-46a1-9411-338a19ceaeb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.529690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145fd280-876b-46a1-9411-338a19ceaeb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.531303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-145fd280-876b-46a1-9411-338a19ceaeb9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.544784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 800/1587] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.544784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.544784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.544784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.570413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] MFA Rules not processed for user `6a153563a2c040de84179a4a2b456030`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:21.572312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:21.580512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 788/1588] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:21.581341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.591501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0439b45-e2cc-4457-a2a7-cb9f8acdf383 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.595976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.596567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.596859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.597117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.597783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.599424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.613411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 789/1589] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.614538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.626058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdca1f81-9fd0-4053-8858-ab1b9e6ff40e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.630815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.631470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.642571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.642571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.642571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.642571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.643013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 801/1590] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:21.643013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.649082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 790/1591] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.650935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.654750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c6f74e4-0988-4878-bd49-b2ba5720fea4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.660967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.661725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.662122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.662496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.662638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d4afe38-3829-4c4f-aa84-0578842f770a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.663620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.665670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.668902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.669895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.669895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.670108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.670653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.672723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.677123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 802/1592] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.678133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.688972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db39e4c1-6922-49c2-8579-dca9bbae0b35 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.689739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 791/1593] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.689739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.698673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.699445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.699790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.700136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.700853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.702353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c0800e-2b2d-422d-a81c-64ac246621d8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.702593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.708818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.709484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.709762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.710199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] PATH_INFO: `/v3/projects/996f7fb4be7b42ceb08488b812087543/users/02e263a6105642c1815e1c59429278f7/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.713453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 803/1594] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:21.714493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.727116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-344d1355-531c-4999-91bd-bc3612130787 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.727620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=996f7fb4be7b42ceb08488b812087543, user_id=02e263a6105642c1815e1c59429278f7, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.730316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.734166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.734765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.735073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.735348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.735975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.737884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.860986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 792/1595] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:21 2022] PUT /identity/v3/projects/996f7fb4be7b42ceb08488b812087543/users/02e263a6105642c1815e1c59429278f7/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 171 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:21.861822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.893617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:21.905516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 804/1596] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/projects => generated 387 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:21.905516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:21.927354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6ee50e-cafd-4b55-88d8-1cafdfd4d2cb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.961167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:21.966940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abfb7f75-8b01-4f18-9914-3d37e6bd2693 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:21.971860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.972463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.972738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.973010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.973640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.975612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.985174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:21.985831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:21.986149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:21.986415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:21.987086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:21.988665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:21.999467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 793/1597] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:22.002996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.009646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13053fca-a988-4f62-b4e1-3aa1b9afd6ca admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.014270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.015193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.015193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.015396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.016000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.017579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.027602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 794/1598] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:21 2022] GET /identity/v3/roles => generated 1420 bytes in 28 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:22.028389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.038362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be6120ba-e77d-433f-8db7-572b57390719 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.043422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.043998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.044275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.044545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.045174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.046731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.057766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 795/1599] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:22 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:22.060397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.067729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1f2ce05-0f41-413e-a4ad-7caaff24d3b0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.072637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-638ec41d-5e31-4040-a255-00ea0a337600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.073210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ec41d-5e31-4040-a255-00ea0a337600 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.073482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ec41d-5e31-4040-a255-00ea0a337600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.073750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-638ec41d-5e31-4040-a255-00ea0a337600 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.074361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638ec41d-5e31-4040-a255-00ea0a337600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.075946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-638ec41d-5e31-4040-a255-00ea0a337600 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.081286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 805/1600] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:21 2022] POST /identity/v3/projects => generated 355 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:22.081286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.087535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 796/1601] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:22 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:22.088667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.089019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.089392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.108228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:22.109555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:22.147231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5086a176-a5c2-4a09-9ea8-b53f0e26c904 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.152527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.153244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.153628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.154006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.154679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.156454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.167118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 806/1602] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:22 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:22.168569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.172100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 797/1603] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:22.172100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.181749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4483589-64d2-4b8f-8f77-43249f397ab3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.186037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2557fb93-1f4d-48f2-9968-3d7d1603037e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.186586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.187307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.187581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.187853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] PATH_INFO: `/v3/projects/73fad530905b485cb7141793f6130a9e/users/e8d3af2b82b641aea11ebf79677a3af1/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.190662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.194812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.194812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.194812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.194812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.194812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.202704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] RBAC: Authorizing `identity:create_grant(project_id=73fad530905b485cb7141793f6130a9e, user_id=e8d3af2b82b641aea11ebf79677a3af1, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.205417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.306211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 807/1604] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/users => generated 300 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:22.307180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.316522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181077e4-0a9a-4758-99e6-879afcf4518b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.324392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 798/1605] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:22 2022] PUT /identity/v3/projects/73fad530905b485cb7141793f6130a9e/users/e8d3af2b82b641aea11ebf79677a3af1/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:22.324392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.338331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:22.381477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.382138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.382416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.382692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] PATH_INFO: `/v3/projects/41cca09950c34cc7baded0eb0a42bb07/users/1dbfb1a3e99c418098c2b02aea93b923/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.385151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a02c11ed-9fa8-45b3-b655-e8fadd94cacd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.389777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.390478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.390713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.391688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] PATH_INFO: `/v3/projects/cb43833648c240a69221229ae62f6846/users/52fdec802b30453d9cd72005159551dc/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.397086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] RBAC: Authorizing `identity:create_grant(project_id=41cca09950c34cc7baded0eb0a42bb07, user_id=1dbfb1a3e99c418098c2b02aea93b923, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.400400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.408099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb43833648c240a69221229ae62f6846, user_id=52fdec802b30453d9cd72005159551dc, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.410698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.483749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 808/1606] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:22 2022] PUT /identity/v3/projects/41cca09950c34cc7baded0eb0a42bb07/users/1dbfb1a3e99c418098c2b02aea93b923/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:22.484538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.517942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 799/1607] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:22 2022] PUT /identity/v3/projects/cb43833648c240a69221229ae62f6846/users/52fdec802b30453d9cd72005159551dc/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:22.518753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.535832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:22.553382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0486ad-ee16-4a28-870a-f69a1e85618f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.584825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da526d9c-c240-44fa-9f6c-596339cb516d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.590650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.591419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.591693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.591963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] PATH_INFO: `/v3/projects/36c1390c5c194a13b1801a9ef3cd1345/users/b56c499872464f3c9e491e0b28114dfd/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.607945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] RBAC: Authorizing `identity:create_grant(project_id=36c1390c5c194a13b1801a9ef3cd1345, user_id=b56c499872464f3c9e491e0b28114dfd, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.609967 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.610569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.610684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.610875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.611146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.611772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.613675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.678238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 800/1608] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:22 2022] PUT /identity/v3/projects/36c1390c5c194a13b1801a9ef3cd1345/users/b56c499872464f3c9e491e0b28114dfd/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:22.679024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.707288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 809/1609] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/users => generated 284 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:22.708037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.747049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4366b002-c80d-4114-afd2-71664caee187 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.751723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a779461-2c5d-4cea-aa9b-425c6cfd6bff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.752384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a779461-2c5d-4cea-aa9b-425c6cfd6bff admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.752666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a779461-2c5d-4cea-aa9b-425c6cfd6bff admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.752939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a779461-2c5d-4cea-aa9b-425c6cfd6bff admin admin] PATH_INFO: `/v3/projects/e98aef08432e4f3889f974b8a2b70f83/users/b491da095fef45c792692be8cd08129b/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.764203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88787f5e-1525-455f-ba81-caadf68b135d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.768386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a779461-2c5d-4cea-aa9b-425c6cfd6bff admin admin] RBAC: Authorizing `identity:create_grant(project_id=e98aef08432e4f3889f974b8a2b70f83, user_id=b491da095fef45c792692be8cd08129b, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.771101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a779461-2c5d-4cea-aa9b-425c6cfd6bff admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.814403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.815080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.815357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.815622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.825264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.826681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.836906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 810/1610] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:22 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:22.837768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:22.860536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 801/1611] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:22 2022] PUT /identity/v3/projects/e98aef08432e4f3889f974b8a2b70f83/users/b491da095fef45c792692be8cd08129b/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 170 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:22.860753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12e88ce-9b2e-4b42-a2e0-859ec25c49c3 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:22.860895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea371e0f-d1f6-4876-87de-e828c6988694 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.860895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea371e0f-d1f6-4876-87de-e828c6988694 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.860895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea371e0f-d1f6-4876-87de-e828c6988694 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.861459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:22.873320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ea371e0f-d1f6-4876-87de-e828c6988694 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:22.875828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ea371e0f-d1f6-4876-87de-e828c6988694 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:22.892924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:22.916246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d3eeb5-3b9c-47dd-91c0-5a5a76132fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:22.916858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d3eeb5-3b9c-47dd-91c0-5a5a76132fd5 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.917250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d3eeb5-3b9c-47dd-91c0-5a5a76132fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.917529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d3eeb5-3b9c-47dd-91c0-5a5a76132fd5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.918167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d3eeb5-3b9c-47dd-91c0-5a5a76132fd5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:22.919798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d3eeb5-3b9c-47dd-91c0-5a5a76132fd5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:22.929932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 811/1612] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:22 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:22.931360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e89742-2df7-45dd-a758-19a0e6bf7b1b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.931663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e89742-2df7-45dd-a758-19a0e6bf7b1b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.932364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88e89742-2df7-45dd-a758-19a0e6bf7b1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.933672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 802/1613] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:22.934963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.935242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.935662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.954583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-88e89742-2df7-45dd-a758-19a0e6bf7b1b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:22.955867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-88e89742-2df7-45dd-a758-19a0e6bf7b1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:22.963698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 812/1614] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:22.964885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74e9b1-24e4-43cc-92f0-7c1579cb9432 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:22.965173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74e9b1-24e4-43cc-92f0-7c1579cb9432 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:22.965445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c74e9b1-24e4-43cc-92f0-7c1579cb9432 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:22.965561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] MFA Rules not processed for user `52fdec802b30453d9cd72005159551dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:22.966718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:22.998911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1c74e9b1-24e4-43cc-92f0-7c1579cb9432 None None] MFA Rules not processed for user `b56c499872464f3c9e491e0b28114dfd`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:23.000583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1c74e9b1-24e4-43cc-92f0-7c1579cb9432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:23.044141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 803/1615] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/auth/tokens => generated 3282 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:23.045114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.060470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8760b2b-c351-4098-a41c-8e32b49ddbfe None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.066381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.067048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.067325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.067596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.068242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.069839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.077219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 813/1616] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:22 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:23.078801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.079107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.079382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.080874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 804/1617] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:23 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:23.081795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.091771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c21a700-ce0b-4e2c-9237-913f670be02d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.096994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.097711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.098057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.098824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] PATH_INFO: `/v3/projects/32a8c5907d594e9ca60da1456bf98eb7/users/21bba6712ee3475fa23410fda871c53a/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.115027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] RBAC: Authorizing `identity:create_grant(project_id=32a8c5907d594e9ca60da1456bf98eb7, user_id=21bba6712ee3475fa23410fda871c53a, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.117746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.120415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] MFA Rules not processed for user `b491da095fef45c792692be8cd08129b`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:23.121580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:23.187338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 814/1618] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:23 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:23.188607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.201141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 805/1619] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:23 2022] PUT /identity/v3/projects/32a8c5907d594e9ca60da1456bf98eb7/users/21bba6712ee3475fa23410fda871c53a/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:23.203968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.268300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:23.270207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7424b4e3-57a9-405d-a6a6-871399d58f1f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.273558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbac607b-f6e9-4a6e-8abf-0277fde235e6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.275553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.277842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.278069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.278440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.278658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.278792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.278999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.279374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.298293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:23.352463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.354711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.363245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.364628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 815/1620] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:23.364774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.365492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.375332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f0f58a-735a-4318-9f3e-d95944f8e25a service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.375472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 806/1621] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:23.375991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.380434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee797fe7-5050-4b44-9884-b832cd1577f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.384876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee797fe7-5050-4b44-9884-b832cd1577f1 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.384876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee797fe7-5050-4b44-9884-b832cd1577f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.384876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee797fe7-5050-4b44-9884-b832cd1577f1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.384876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee797fe7-5050-4b44-9884-b832cd1577f1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.384876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee797fe7-5050-4b44-9884-b832cd1577f1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.385969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32466812-ae57-4769-8266-ca7df50f8a50 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.391722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.392875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.393111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.393380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] PATH_INFO: `/v3/projects/c41d2fe78f3c4f3cbb71565656e3548c/users/98495404c1dd4416b9278fedfc9c1ae4/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.409588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] RBAC: Authorizing `identity:create_grant(project_id=c41d2fe78f3c4f3cbb71565656e3548c, user_id=98495404c1dd4416b9278fedfc9c1ae4, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.412278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.484249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 816/1622] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:23 2022] POST /identity/v3/projects => generated 403 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:23.485788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d6a8db-849c-48fe-8691-b3d625ba6dac None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.485979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d6a8db-849c-48fe-8691-b3d625ba6dac None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.486114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 807/1623] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:23 2022] PUT /identity/v3/projects/c41d2fe78f3c4f3cbb71565656e3548c/users/98495404c1dd4416b9278fedfc9c1ae4/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:23.489215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d6a8db-849c-48fe-8691-b3d625ba6dac None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.489347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.492252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 817/1624] 10.222.0.64 () {54 vars in 961 bytes} [Tue Sep 27 16:36:23 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:36:23.494287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d6a8db-849c-48fe-8691-b3d625ba6dac None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.574500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d6a8db-849c-48fe-8691-b3d625ba6dac None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.582352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0f7c14e-c263-4a63-99cb-587ff3fea7ed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.585983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.586476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.586854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.587142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.587265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.587491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.587963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.588833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.589595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.589811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.591505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.591882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.604629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 818/1625] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:23 2022] GET /identity/v3/roles => generated 1420 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:23.606477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.633060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dd1d86f-3343-4723-8b99-56585a7899a5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.641365 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.642377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.642921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.643457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.644561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.646881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.690395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 808/1626] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:23 2022] POST /identity/v3/projects => generated 387 bytes in 204 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:23.692019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.763179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38da29da-1879-4ecc-834d-2149f49d7f10 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.767981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fda9b766-2578-40a4-b75c-847f72920dac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.768591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda9b766-2578-40a4-b75c-847f72920dac admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.768872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda9b766-2578-40a4-b75c-847f72920dac admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.769146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fda9b766-2578-40a4-b75c-847f72920dac admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.769854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda9b766-2578-40a4-b75c-847f72920dac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.771492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fda9b766-2578-40a4-b75c-847f72920dac admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.782365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 809/1627] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:23 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:23.783745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.784155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.784438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.788848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 819/1628] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:23 2022] POST /identity/v3/users => generated 308 bytes in 184 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:23.789668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.800674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c546db98-a79a-4cfe-8bb8-fac064b994a4 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.804517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:23.808873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.809556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.809879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.810161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] PATH_INFO: `/v3/projects/32a8c5907d594e9ca60da1456bf98eb7/users/21bba6712ee3475fa23410fda871c53a/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.813602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32a8c5907d594e9ca60da1456bf98eb7, user_id=21bba6712ee3475fa23410fda871c53a, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.818377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.833862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 820/1629] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:23 2022] PUT /identity/v3/projects/32a8c5907d594e9ca60da1456bf98eb7/users/21bba6712ee3475fa23410fda871c53a/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:23.834936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.848804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:23.850161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:23.889610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:23.914755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaedde6-4e2f-42a0-9ff7-d7052f60ec93 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.920679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df07245c-6cf3-4b0b-8fbf-53324c1a06b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.921434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df07245c-6cf3-4b0b-8fbf-53324c1a06b6 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.921603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df07245c-6cf3-4b0b-8fbf-53324c1a06b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.922479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df07245c-6cf3-4b0b-8fbf-53324c1a06b6 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.923146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df07245c-6cf3-4b0b-8fbf-53324c1a06b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.925109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df07245c-6cf3-4b0b-8fbf-53324c1a06b6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.933541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 810/1630] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Sep 27 16:36:23 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 150 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:23.934532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:23.950127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c2f1620-faee-4536-9046-6935c35cbfd4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:23.955325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:23.956314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:23.956314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:23.956569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] PATH_INFO: `/v3/projects/c41d2fe78f3c4f3cbb71565656e3548c/users/98495404c1dd4416b9278fedfc9c1ae4/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:23.958945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c41d2fe78f3c4f3cbb71565656e3548c, user_id=98495404c1dd4416b9278fedfc9c1ae4, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:23.961821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:23.983507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 811/1631] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:23 2022] PUT /identity/v3/projects/c41d2fe78f3c4f3cbb71565656e3548c/users/98495404c1dd4416b9278fedfc9c1ae4/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:23.983507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.020224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 821/1632] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:23 2022] POST /identity/v3/users => generated 294 bytes in 186 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:24.022412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252a3d7-3777-42e9-b4d8-c08dd3052f89 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.022698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252a3d7-3777-42e9-b4d8-c08dd3052f89 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.022994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4252a3d7-3777-42e9-b4d8-c08dd3052f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.057385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4252a3d7-3777-42e9-b4d8-c08dd3052f89 None None] MFA Rules not processed for user `21bba6712ee3475fa23410fda871c53a`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:24.058962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4252a3d7-3777-42e9-b4d8-c08dd3052f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:24.066751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06262c6e-d4e0-4f4f-b256-8d544e8e2104 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.071599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.072271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.072690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.073134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.073942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.075740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.085881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 812/1633] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:23 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:24.086804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.134935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 822/1634] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:24 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:24.137045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.137313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.137577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.169431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] MFA Rules not processed for user `98495404c1dd4416b9278fedfc9c1ae4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:24.170917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:24.174100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01d87b7a-b349-4538-a783-0007967ec67f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.182639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.183333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.183629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.183884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.189040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:24.258458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.263286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.277589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 813/1635] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:24 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 191 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:24.278637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.290856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74f44939-5cbb-4c6c-a37c-8501496dba33 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.298199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.299415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.300170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.300904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.302058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.304095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.309862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 823/1636] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:24 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:24.311212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.326472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78094e13-4ca9-41db-9e71-840f6d935a54 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.336576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.336576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.336576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.336940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 814/1637] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:24 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:24.337045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] PATH_INFO: `/v3/projects/0ca2404a137443e0868daafb9a74e77a/users/5e95b812ab9c49d9a4013bc5d68d02b1/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.337131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.358753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ca2404a137443e0868daafb9a74e77a, user_id=5e95b812ab9c49d9a4013bc5d68d02b1, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.362495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.405826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d32da9d-bf74-480e-8c7d-256ae1d3a5f6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.409494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.410201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.410546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.410929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.420904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.423904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.432751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 815/1638] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:24 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:24.435328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.447136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b060a905-d7f0-4ea2-9ea3-aebcd1b6a768 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.452272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.453233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.453233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.453463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.483333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.487038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.499996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 816/1639] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:24 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:24.511829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.516426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 824/1640] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:24 2022] PUT /identity/v3/projects/0ca2404a137443e0868daafb9a74e77a/users/5e95b812ab9c49d9a4013bc5d68d02b1/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:24.525122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.538569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bf218ca-c956-4b39-9f21-337499fe05f1 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.627565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:24.663347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0694d70-ec23-4b72-b026-4a28bf48c4a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.664254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0694d70-ec23-4b72-b026-4a28bf48c4a2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.664634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0694d70-ec23-4b72-b026-4a28bf48c4a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.665006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0694d70-ec23-4b72-b026-4a28bf48c4a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.665755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0694d70-ec23-4b72-b026-4a28bf48c4a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.667735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0694d70-ec23-4b72-b026-4a28bf48c4a2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.695675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 817/1641] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:24 2022] GET /identity/v3/roles => generated 1420 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:24.698297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d4708f-3430-48e4-a1cb-08c836c441b9 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.698297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d4708f-3430-48e4-a1cb-08c836c441b9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.698704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40d4708f-3430-48e4-a1cb-08c836c441b9 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.700813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 818/1642] 10.222.0.64 () {54 vars in 912 bytes} [Tue Sep 27 16:36:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:36:24.701658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d4708f-3430-48e4-a1cb-08c836c441b9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.720394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3e11e51-e720-46ee-896e-0c992cb355f7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.731443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40d4708f-3430-48e4-a1cb-08c836c441b9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:24.733576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c31e92f9-2e7c-4585-b8d9-f9f711eb480b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.735275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31e92f9-2e7c-4585-b8d9-f9f711eb480b admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.758406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31e92f9-2e7c-4585-b8d9-f9f711eb480b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.758406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c31e92f9-2e7c-4585-b8d9-f9f711eb480b admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.758406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31e92f9-2e7c-4585-b8d9-f9f711eb480b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.759026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:24.759026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.759026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.759026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] PATH_INFO: `/v3/projects/3cfe269f5cb749959937adef4c17915a/users/baa6149227c842a1907adb8950ed1a2e/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.759475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c31e92f9-2e7c-4585-b8d9-f9f711eb480b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.781917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 825/1643] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:24 2022] GET /identity/v3/roles => generated 1420 bytes in 262 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:36:24.784832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cfe269f5cb749959937adef4c17915a, user_id=baa6149227c842a1907adb8950ed1a2e, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:24.785548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d61240-d5dc-425e-a01c-818cade20621 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.785863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d61240-d5dc-425e-a01c-818cade20621 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.786125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60d61240-d5dc-425e-a01c-818cade20621 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:24.798852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:24.831971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-60d61240-d5dc-425e-a01c-818cade20621 None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:24.833219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-60d61240-d5dc-425e-a01c-818cade20621 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:24.934521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 819/1644] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:24 2022] PUT /identity/v3/projects/3cfe269f5cb749959937adef4c17915a/users/baa6149227c842a1907adb8950ed1a2e/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:24.934521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:24.946025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 826/1645] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:36:24 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:24.946025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:24.946025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:24.946025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.231462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:25.231688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] MFA Rules not processed for user `baa6149227c842a1907adb8950ed1a2e`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:25.231688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:25.231908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4335084c-3fba-4cac-b327-b054529b7060 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:25.231908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:25.231908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:25.231908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:25.232643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] PATH_INFO: `/v3/projects/0ca2404a137443e0868daafb9a74e77a/users/5e95b812ab9c49d9a4013bc5d68d02b1/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.232643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ca2404a137443e0868daafb9a74e77a, user_id=5e95b812ab9c49d9a4013bc5d68d02b1, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:25.232643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:25.241447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 827/1646] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:24 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:25.241447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:25.241917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 820/1647] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:24 2022] PUT /identity/v3/projects/0ca2404a137443e0868daafb9a74e77a/users/5e95b812ab9c49d9a4013bc5d68d02b1/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:25.241917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:25.242130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83db2001-bed1-4b98-b12b-9c196a094598 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:25.242288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d523f8b-c848-408e-947d-9688919def71 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:25.242288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:25.242288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:25.242288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:25.242556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.246886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75cd8be6-d266-4759-9d2e-020c3ba807d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:25.246886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cd8be6-d266-4759-9d2e-020c3ba807d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:25.246886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cd8be6-d266-4759-9d2e-020c3ba807d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:25.246886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75cd8be6-d266-4759-9d2e-020c3ba807d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.318218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:25.319725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:25.334401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 821/1648] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:25 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 235 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:25.335274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:25.371200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cd8be6-d266-4759-9d2e-020c3ba807d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:25.372733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75cd8be6-d266-4759-9d2e-020c3ba807d0 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:25.391861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 828/1649] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:25.394034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:25.394464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:25.394925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.431715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] MFA Rules not processed for user `5e95b812ab9c49d9a4013bc5d68d02b1`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:25.436515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:25.461702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc277a9b-a00a-4f3e-a2e3-e1499b71f322 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:25.468660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:25.483396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:25.484019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:25.484939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:25.485268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.486995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:25.493392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:25.591742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 829/1650] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:25 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:25.771960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:25.778912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 822/1651] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:25 2022] POST /identity/v3/projects => generated 385 bytes in 439 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:25.799655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9ac9399-bbf3-4784-9ee9-d751ac7360e1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:25.914873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:25.922156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:25.922156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:25.922156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:25.922156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:25.924721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:25.987028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:26.169558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 830/1652] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:25 2022] POST /identity/v3/users => generated 299 bytes in 398 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:26.173617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:26.188508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16565def-e90a-4c5c-a730-2cc7b6b2a6ec admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:26.198990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:26.200588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:26.200588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:26.200588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:26.201983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:26.203883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:26.216171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 823/1653] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:26 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:26.221638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:26.222917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:26.243776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf59d7ca-66cc-470d-a6ec-3dcc808cf859 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:26.259316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:26.260116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:26.260457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:26.261015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] PATH_INFO: `/v3/projects/08d0c840081b419ab17893b5520a7bde/users/55bfd6d2d0a14bdc909d5c407ef56c3b/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:26.325682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] RBAC: Authorizing `identity:create_grant(project_id=08d0c840081b419ab17893b5520a7bde, user_id=55bfd6d2d0a14bdc909d5c407ef56c3b, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:26.334959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:26.477312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 831/1654] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:26 2022] PUT /identity/v3/projects/08d0c840081b419ab17893b5520a7bde/users/55bfd6d2d0a14bdc909d5c407ef56c3b/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:26.478035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:26.487758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:26.499269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22d6ad7e-60f5-4a0c-ab35-2055805161ed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:26.593137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bfabbae-033e-46c1-8b0e-84b7021fff9d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:26.602075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:26.605448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:26.605448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:26.605448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:26.642161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:26.642321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:26.642466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:26.643044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:26.643618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:26.645024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:26.670819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 824/1655] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:26 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:36:26.672112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:26.696006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00d1988e-d01a-400a-8959-cc636af502bd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:26.706947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae389e42-1999-4952-9728-339048c982af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:26.707662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae389e42-1999-4952-9728-339048c982af admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:26.707889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae389e42-1999-4952-9728-339048c982af admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:26.708160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae389e42-1999-4952-9728-339048c982af admin admin] PATH_INFO: `/v3/projects/08d0c840081b419ab17893b5520a7bde/users/55bfd6d2d0a14bdc909d5c407ef56c3b/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:26.710329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae389e42-1999-4952-9728-339048c982af admin admin] RBAC: Authorizing `identity:create_grant(project_id=08d0c840081b419ab17893b5520a7bde, user_id=55bfd6d2d0a14bdc909d5c407ef56c3b, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:26.719240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae389e42-1999-4952-9728-339048c982af admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:26.755970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 825/1656] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:26 2022] PUT /identity/v3/projects/08d0c840081b419ab17893b5520a7bde/users/55bfd6d2d0a14bdc909d5c407ef56c3b/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 82 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:26.766437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45143aa9-66ac-4c42-b18c-0a869ffcaf8a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:26.766845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45143aa9-66ac-4c42-b18c-0a869ffcaf8a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:26.767196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45143aa9-66ac-4c42-b18c-0a869ffcaf8a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:26.794418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:26.796206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:26.824905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 832/1657] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:26 2022] GET /identity/v3/auth/tokens => generated 3347 bytes in 338 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:26.851543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-45143aa9-66ac-4c42-b18c-0a869ffcaf8a None None] MFA Rules not processed for user `55bfd6d2d0a14bdc909d5c407ef56c3b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:26.859427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-45143aa9-66ac-4c42-b18c-0a869ffcaf8a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:27.081772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 826/1658] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:26 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 316 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:27.373989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:27.388051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0722b6a2-f1fd-4405-a032-5a8bf2a91468 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:27.424128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:27.459992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e876fb9-9e09-4e27-8c87-2ba2c9e20c49 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:27.460712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e876fb9-9e09-4e27-8c87-2ba2c9e20c49 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:27.461106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e876fb9-9e09-4e27-8c87-2ba2c9e20c49 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:27.461511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e876fb9-9e09-4e27-8c87-2ba2c9e20c49 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:27.497391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e876fb9-9e09-4e27-8c87-2ba2c9e20c49 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:27.500659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e876fb9-9e09-4e27-8c87-2ba2c9e20c49 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:27.547630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 833/1659] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:27 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 174 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:27.574703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c2867b-826e-49f3-ac32-707ff1671495 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:27.575736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c2867b-826e-49f3-ac32-707ff1671495 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:27.576267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c2867b-826e-49f3-ac32-707ff1671495 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:27.578495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 827/1660] 10.222.0.64 () {54 vars in 961 bytes} [Tue Sep 27 16:36:27 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:36:27.584884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53ffb2-e1a7-4ae1-a0bd-c259f27461fb None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:27.585558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53ffb2-e1a7-4ae1-a0bd-c259f27461fb None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:27.586026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e53ffb2-e1a7-4ae1-a0bd-c259f27461fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:27.624343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4e53ffb2-e1a7-4ae1-a0bd-c259f27461fb None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:27.630566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4e53ffb2-e1a7-4ae1-a0bd-c259f27461fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:27.835306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 834/1661] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Sep 27 16:36:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 252 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:27.848423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe98253-17b2-4147-8509-5558f5c499ec None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:27.848676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe98253-17b2-4147-8509-5558f5c499ec None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:27.848942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe98253-17b2-4147-8509-5558f5c499ec None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:27.852363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 828/1662] 10.222.0.64 () {54 vars in 912 bytes} [Tue Sep 27 16:36:27 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:36:27.865555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:27.866063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:27.866669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:27.899800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe98253-17b2-4147-8509-5558f5c499ec None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:27.991315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:27.995707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:28.061002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 835/1663] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:36:27 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:28.082691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:28.216729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe98253-17b2-4147-8509-5558f5c499ec None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:28.230088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:28.230760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:28.231064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:28.231337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] PATH_INFO: `/v3/users/52fdec802b30453d9cd72005159551dc` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:28.233212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] RBAC: Authorizing `identity:delete_user(user_id=52fdec802b30453d9cd72005159551dc)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:28.235687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: WARNING py.warnings [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:36:28.235687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: warnings.warn(msg) Sep 27 16:36:28.235687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: Sep 27 16:36:28.236930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:28.329946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:28.350177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b874849-0620-4348-a3bb-f814b919e9cc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:28.360573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2687cd66-59f9-4212-9893-b44f9d77415a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:28.361373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2687cd66-59f9-4212-9893-b44f9d77415a service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:28.361612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2687cd66-59f9-4212-9893-b44f9d77415a service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:28.361882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2687cd66-59f9-4212-9893-b44f9d77415a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:28.400354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2687cd66-59f9-4212-9893-b44f9d77415a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:28.405158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2687cd66-59f9-4212-9893-b44f9d77415a service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:28.448907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 836/1664] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:28 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:28.542043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '52fdec802b30453d9cd72005159551dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:28.569728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '52fdec802b30453d9cd72005159551dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:28.572700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '52fdec802b30453d9cd72005159551dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:28.573329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '52fdec802b30453d9cd72005159551dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:28.677056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '52fdec802b30453d9cd72005159551dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:28.779020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:28.779545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:28.779545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:28.782434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 829/1665] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:27 2022] DELETE /identity/v3/users/52fdec802b30453d9cd72005159551dc => generated 0 bytes in 874 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:28.828519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:28.829746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:28.932473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 837/1666] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:28.943600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:28.945153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:29.141342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e31ba0e-36b4-431a-99f1-f0df6a21b74f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:29.158144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:29.159099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:29.159531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:29.163744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:29.197865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:29.199550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:29.227949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 830/1667] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 285 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:30.448251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:30.450652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:30.484831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7253a048-69fb-4910-afab-5586ede8e8aa None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:30.510871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:30.516135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:30.517622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:30.518133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] PATH_INFO: `/v3/users/8c3fb5ebdd5746a6b3bd667e5338c42e` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:30.523852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c3fb5ebdd5746a6b3bd667e5338c42e)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:30.530268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:30.785122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:30.787201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:30.817323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:30.856793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c3fb5ebdd5746a6b3bd667e5338c42e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:30.857607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c3fb5ebdd5746a6b3bd667e5338c42e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:30.868172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c3fb5ebdd5746a6b3bd667e5338c42e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:30.869629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c3fb5ebdd5746a6b3bd667e5338c42e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:30.936413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:30.957271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258a1fdc-d4a7-4691-b23d-d7f80a208dce service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:30.971531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:30.972181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:30.972642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:30.973564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:30.978722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:30.986879 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4fab4f51-cc25-4508-9242-fc848d9c1cbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c3fb5ebdd5746a6b3bd667e5338c42e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:31.022754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 838/1668] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:30 2022] DELETE /identity/v3/users/8c3fb5ebdd5746a6b3bd667e5338c42e => generated 0 bytes in 576 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:31.028298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5ce0e-073f-409d-87da-e902f91db7c6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:31.028894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5ce0e-073f-409d-87da-e902f91db7c6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:31.029324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03d5ce0e-073f-409d-87da-e902f91db7c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:31.079019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-03d5ce0e-073f-409d-87da-e902f91db7c6 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:31.080345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-03d5ce0e-073f-409d-87da-e902f91db7c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:31.170412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:31.171281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:31.211460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 831/1669] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:30 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:31.235989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 839/1670] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:31.248809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:31.251220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:31.270638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7098ce-d974-4c11-ae70-bd1c19528a54 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:31.427684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b9fedb2-d01c-4e64-b849-df2023c81e33 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:31.428239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fedb2-d01c-4e64-b849-df2023c81e33 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:31.428519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fedb2-d01c-4e64-b849-df2023c81e33 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:31.428788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b9fedb2-d01c-4e64-b849-df2023c81e33 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:31.461369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9fedb2-d01c-4e64-b849-df2023c81e33 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:31.467019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b9fedb2-d01c-4e64-b849-df2023c81e33 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:31.504421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 832/1671] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 256 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:31.654992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:31.656014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:31.656014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:31.712377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] MFA Rules not processed for user `02e263a6105642c1815e1c59429278f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:31.713949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:31.842174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 840/1672] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:31.851638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:31.852552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:31.853036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:31.874947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] MFA Rules not processed for user `02e263a6105642c1815e1c59429278f7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:31.876412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:31.876932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:31.879046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:31.885794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 833/1673] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:31 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:31.903357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:31.905333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:31.906044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da7ff51-6435-4c13-9cd1-bb2b50e193ef None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:31.912312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] PATH_INFO: `/v3/projects/cb43833648c240a69221229ae62f6846` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb43833648c240a69221229ae62f6846)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: WARNING py.warnings [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: warnings.warn(msg) Sep 27 16:36:31.916595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: Sep 27 16:36:31.919319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:32.141923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb43833648c240a69221229ae62f6846', 'request_id': 'req-b0bfaa6f-7e5f-4286-a481-755807a69897'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.156890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67e7169-f118-4ff7-96ed-0248850b4f8f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:32.248451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb43833648c240a69221229ae62f6846', 'request_id': 'req-b0bfaa6f-7e5f-4286-a481-755807a69897'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.282346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb43833648c240a69221229ae62f6846', 'request_id': 'req-b0bfaa6f-7e5f-4286-a481-755807a69897'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.318404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] The token cache is being invalidate because project cb43833648c240a69221229ae62f6846 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:32.323745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.326815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.330587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 841/1674] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:31 2022] DELETE /identity/v3/projects/cb43833648c240a69221229ae62f6846 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:32.334581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:32.339735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:32.352722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:32.356163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:32.356632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:32.357008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:32.361145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:32.433243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:32.499552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0bfaa6f-7e5f-4286-a481-755807a69897 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:32.504567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:32.506066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:32.506590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:32.507253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:32.507548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:32.507911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] PATH_INFO: `/v3/users/5e95b812ab9c49d9a4013bc5d68d02b1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:32.514732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5e95b812ab9c49d9a4013bc5d68d02b1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:32.519940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:32.523266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 834/1675] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:31 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 621 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:36:32.524578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:32.527323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:32.554035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e21496-b5ae-4218-88a5-e46d6a830538 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:32.566063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:32.566969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:32.567403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:32.568765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] PATH_INFO: `/v3/projects/a1ae034405f34841bd96ab9c70711049` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:32.570400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1ae034405f34841bd96ab9c70711049)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:32.573754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:32.701064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5e95b812ab9c49d9a4013bc5d68d02b1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.702420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5e95b812ab9c49d9a4013bc5d68d02b1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.720903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5e95b812ab9c49d9a4013bc5d68d02b1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.724810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5e95b812ab9c49d9a4013bc5d68d02b1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.735554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1ae034405f34841bd96ab9c70711049', 'request_id': 'req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.782462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1ae034405f34841bd96ab9c70711049', 'request_id': 'req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.824569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5e95b812ab9c49d9a4013bc5d68d02b1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.861117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 842/1676] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:32 2022] DELETE /identity/v3/users/5e95b812ab9c49d9a4013bc5d68d02b1 => generated 0 bytes in 526 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:32.917467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1ae034405f34841bd96ab9c70711049', 'request_id': 'req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.943181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] The token cache is being invalidate because project a1ae034405f34841bd96ab9c70711049 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:32.947783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.953369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1db0d32-cbc1-4407-94b8-4c08f49d1ccb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:32.964781 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 835/1677] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:32 2022] DELETE /identity/v3/projects/a1ae034405f34841bd96ab9c70711049 => generated 0 bytes in 437 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:32.968041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:32.969977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:33.143722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e46cc5-a9db-4edd-b698-ff02c3631cc1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:33.156893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:33.157445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:33.157725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:33.158003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] PATH_INFO: `/v3/users/21bba6712ee3475fa23410fda871c53a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:33.161714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] RBAC: Authorizing `identity:delete_user(user_id=21bba6712ee3475fa23410fda871c53a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:33.165540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:33.311523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6da0a2-06a1-4bff-a8d3-dfbae1bd421b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:33.313855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6da0a2-06a1-4bff-a8d3-dfbae1bd421b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:33.313855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b6da0a2-06a1-4bff-a8d3-dfbae1bd421b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:33.313855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 836/1678] 10.222.0.64 () {54 vars in 912 bytes} [Tue Sep 27 16:36:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:36:33.321064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:33.321651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:33.322592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:33.427907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] MFA Rules not processed for user `28f0f9fef2f54e56a7a92d077c25d2b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:33.432424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:33.506764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21bba6712ee3475fa23410fda871c53a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:33.510964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21bba6712ee3475fa23410fda871c53a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:33.527695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21bba6712ee3475fa23410fda871c53a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:33.530338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21bba6712ee3475fa23410fda871c53a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:33.532956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 837/1679] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:36:33 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:33.547361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:33.552035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:33.585588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a7530eb-3f39-468a-baf9-0c049c637bca None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:33.592401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:33.593452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:33.594069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:33.594527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:33.616876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:33.620379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21bba6712ee3475fa23410fda871c53a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:33.621818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:33.653874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 843/1680] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:32 2022] DELETE /identity/v3/users/21bba6712ee3475fa23410fda871c53a => generated 0 bytes in 685 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:33.662910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 838/1681] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:33 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:33.757649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:34.483676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:34.742186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:34.752124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:34.921268 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-481b3f94-a940-453f-9ad8-2d9099eeaa19 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:34.939869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:34.943038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:34.945272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:34.946693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] PATH_INFO: `/v3/projects/32a8c5907d594e9ca60da1456bf98eb7` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:34.952913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=32a8c5907d594e9ca60da1456bf98eb7)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:34.960807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:34.995430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:34.999426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:34.999426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:35.206367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f01cdb-8158-4df9-9acf-81a063e03556 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:35.280684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32a8c5907d594e9ca60da1456bf98eb7', 'request_id': 'req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:35.339291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:35.339975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:35.340443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:35.341035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:35.346023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:35.376719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:35.378508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-747f8a03-6bcd-4bb9-9b19-72d53739a5ac service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:35.380493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32a8c5907d594e9ca60da1456bf98eb7', 'request_id': 'req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:35.391139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32a8c5907d594e9ca60da1456bf98eb7', 'request_id': 'req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:35.403788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 839/1682] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:34 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:35.426768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] The token cache is being invalidate because project 32a8c5907d594e9ca60da1456bf98eb7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:35.427226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:35.432112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9456655d-3e4f-45e8-81b7-9ef8a9dce9ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:35.447913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:35.448362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:35.448990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:35.458369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 844/1683] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:34 2022] DELETE /identity/v3/projects/32a8c5907d594e9ca60da1456bf98eb7 => generated 0 bytes in 698 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:35.499294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:35.500632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:35.625860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 840/1684] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:35.635664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:35.635954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:35.636219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:35.652632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:35.659307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:35.676688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:35.677944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:35.702052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a745c1ce-0493-414c-b398-0939002b45f5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:35.702161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 845/1685] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:35.704586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:35.721962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:35.722752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:35.729654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:35.730060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] PATH_INFO: `/v3/projects/0ca2404a137443e0868daafb9a74e77a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:35.731776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ca2404a137443e0868daafb9a74e77a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:35.734572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:35.743709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39cea800-9fe0-43c2-9dad-b14855d1abfe None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:35.792703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:35.796480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:35.796813 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:35.797052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:35.800944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:35.802824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:35.810194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:35.867896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 846/1686] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:35.874534 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:35.914950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a65cf91f-27ec-47ba-a8f0-907d6b41a6fb admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:35.932037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ca2404a137443e0868daafb9a74e77a', 'request_id': 'req-da73f702-a700-43fd-9402-bb6bdc764db7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:35.946991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ca2404a137443e0868daafb9a74e77a', 'request_id': 'req-da73f702-a700-43fd-9402-bb6bdc764db7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:36.023455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ca2404a137443e0868daafb9a74e77a', 'request_id': 'req-da73f702-a700-43fd-9402-bb6bdc764db7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:36.025650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.029924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.029924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.029924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.029924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.032915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.044588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] The token cache is being invalidate because project 0ca2404a137443e0868daafb9a74e77a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:36.044891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:36.045671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-da73f702-a700-43fd-9402-bb6bdc764db7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:36.060579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 841/1687] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:35 2022] DELETE /identity/v3/projects/0ca2404a137443e0868daafb9a74e77a => generated 0 bytes in 411 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:36.065435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c37c8-ae40-4e8c-a34d-91eb3cdb3eae None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.066013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c37c8-ae40-4e8c-a34d-91eb3cdb3eae None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.069939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65c37c8-ae40-4e8c-a34d-91eb3cdb3eae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.099157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d65c37c8-ae40-4e8c-a34d-91eb3cdb3eae None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:36.100700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d65c37c8-ae40-4e8c-a34d-91eb3cdb3eae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:36.132092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 842/1688] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:36.141268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.141806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.142291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.171955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 847/1689] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:35 2022] POST /identity/v3/projects => generated 373 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:36.176558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.178794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:36.204131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:36.205799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:36.360534 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04a6eb59-e236-4d3f-892b-5bbd741f95cd admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.372363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.372993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.373603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.376983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.379664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.383682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.414133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 843/1690] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 274 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:36.418104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.460969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318cf93c-15a6-4628-811f-ebac89125fe9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.470144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.471442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.477723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.477723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.477723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:36.477723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.477723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.491814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 844/1691] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:36:36.495873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.512232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d170a51-38dd-4d6c-a521-0bcfb81d1ee6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.513981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 848/1692] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:36 2022] POST /identity/v3/users => generated 287 bytes in 338 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:36.518012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.530604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.531764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031ccf6c-af0b-4ca6-aa1f-c722e8ace644 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.535432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.535863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.536291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.537081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.539614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.545307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.546252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.546541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.546843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.547950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.550266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.575892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 849/1693] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:36 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:36:36.577128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.621465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1b68bf2-20e4-4ff3-bf14-df47e0a6ab77 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.634133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.646057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.646057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.646057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.646057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.646057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.654693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 850/1694] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:36 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:36.688230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.711529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 845/1695] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:36 2022] POST /identity/v3/projects => generated 385 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:36.715282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:36.719475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c1c93ff-5d53-49cb-820b-24381fdc039d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.866975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3151f18-9065-4f42-b72b-34d2559758ae admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:36.867801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.868486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.868894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.869364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] PATH_INFO: `/v3/projects/1707bf798d9f4731ba5a5aa8edb97d19/users/299d4e120b76468fbd183f9339549f83/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.886377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3bfca99-6ae9-46d8-9dce-a953bf3a8f02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:36.887034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bfca99-6ae9-46d8-9dce-a953bf3a8f02 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:36.887388 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bfca99-6ae9-46d8-9dce-a953bf3a8f02 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:36.887669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3bfca99-6ae9-46d8-9dce-a953bf3a8f02 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:36.888723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bfca99-6ae9-46d8-9dce-a953bf3a8f02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.892464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3bfca99-6ae9-46d8-9dce-a953bf3a8f02 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.907121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1707bf798d9f4731ba5a5aa8edb97d19, user_id=299d4e120b76468fbd183f9339549f83, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:36.909867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:36.997974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 851/1696] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:36 2022] PUT /identity/v3/projects/1707bf798d9f4731ba5a5aa8edb97d19/users/299d4e120b76468fbd183f9339549f83/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 312 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:36.998429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 846/1697] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:36 2022] POST /identity/v3/users => generated 293 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:37.001491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:37.009256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:37.010219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:37.010641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:37.137545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] MFA Rules not processed for user `299d4e120b76468fbd183f9339549f83`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:37.138807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:37.207532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c01ee6c-fdce-4995-95d7-d537dd52a198 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:37.221602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:37.223667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:37.224050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:37.224298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:37.224960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:37.227712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:37.281822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 852/1698] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:36 2022] GET /identity/v3/roles => generated 1420 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:36:37.282847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:37.295611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:37.339402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 847/1699] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:37 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:37.340236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b524cf-5dab-4281-b65a-48b6af94c9b3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:37.340331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:37.342943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:37.347281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:37.348631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:37.350310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] PATH_INFO: `/v3/users/b56c499872464f3c9e491e0b28114dfd` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:37.351809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd88892-048d-4032-9521-b686e90f8bf4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:37.352684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b56c499872464f3c9e491e0b28114dfd)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:37.359046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:37.378319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:37.382127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:37.383643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:37.391532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:37.391955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:37.394860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:37.401684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:37.439432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 848/1700] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:37 2022] GET /identity/v3/roles => generated 1420 bytes in 113 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:37.440589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:37.442674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:37.614433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c883e5e-4daa-441e-ba75-97c125bf5ae4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:37.620356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:37.620992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:37.621270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:37.621544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:37.636064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:37.636064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:37.672620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 849/1701] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:37 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:37.675150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:37.731733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-164f8874-4594-480f-a6bd-0cc476642e41 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:37.760051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:37.760051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:37.760051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:37.760051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] PATH_INFO: `/v3/projects/f777db7c495e4c70a575eb9ff6edcbe9/users/b53329615627482792c4817e98f118f2/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:37.806865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f777db7c495e4c70a575eb9ff6edcbe9, user_id=b53329615627482792c4817e98f118f2, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:37.810107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:37.840625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b56c499872464f3c9e491e0b28114dfd'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:37.842484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b56c499872464f3c9e491e0b28114dfd'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:37.860384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b56c499872464f3c9e491e0b28114dfd'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:37.864863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b56c499872464f3c9e491e0b28114dfd'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:37.920221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 850/1702] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:37 2022] PUT /identity/v3/projects/f777db7c495e4c70a575eb9ff6edcbe9/users/b53329615627482792c4817e98f118f2/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 237 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:37.920633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:37.922409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:37.925102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:37.998351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b12d6ac9-5060-491c-8026-768d2ad7fe0d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b56c499872464f3c9e491e0b28114dfd'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:38.024979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56c73f85-d8f4-4b5f-95fd-5dfa25d14293 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:38.034632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:38.035560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.043135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.043135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] PATH_INFO: `/v3/users/299d4e120b76468fbd183f9339549f83` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.043135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] RBAC: Authorizing `identity:delete_user(user_id=299d4e120b76468fbd183f9339549f83)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:38.043135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:38.044292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 853/1703] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:37 2022] DELETE /identity/v3/users/b56c499872464f3c9e491e0b28114dfd => generated 0 bytes in 757 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:38.046281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a3597-f9bb-48d9-bd04-d0478fa63a4e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.047728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a3597-f9bb-48d9-bd04-d0478fa63a4e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.048833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2a3597-f9bb-48d9-bd04-d0478fa63a4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.091035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ff2a3597-f9bb-48d9-bd04-d0478fa63a4e None None] MFA Rules not processed for user `1dbfb1a3e99c418098c2b02aea93b923`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:38.092755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ff2a3597-f9bb-48d9-bd04-d0478fa63a4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.231061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 854/1704] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:38 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 186 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:38.233104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633d3e5b-0b1e-4488-92b7-f3147ef43364 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.233458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633d3e5b-0b1e-4488-92b7-f3147ef43364 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.234071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-633d3e5b-0b1e-4488-92b7-f3147ef43364 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.283450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-633d3e5b-0b1e-4488-92b7-f3147ef43364 None None] MFA Rules not processed for user `b53329615627482792c4817e98f118f2`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:38.285280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-633d3e5b-0b1e-4488-92b7-f3147ef43364 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.302793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '299d4e120b76468fbd183f9339549f83'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:38.309278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:38.346529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '299d4e120b76468fbd183f9339549f83'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:38.347324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '299d4e120b76468fbd183f9339549f83'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:38.347907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '299d4e120b76468fbd183f9339549f83'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:38.369867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 855/1705] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:38 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:38.369867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.369867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.369867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.395895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:38.397145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.472753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '299d4e120b76468fbd183f9339549f83'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:38.532634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 851/1706] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:37 2022] DELETE /identity/v3/users/299d4e120b76468fbd183f9339549f83 => generated 0 bytes in 622 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:38.533395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 856/1707] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:38.535384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:38.536405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:38.539557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.544344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.567326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10941e49-6fc1-4bc9-bb10-df402398abcc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:38.722543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0c50005-ceae-4452-ad8c-b66036bd9965 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:38.748030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:38.748030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.748030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.748030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.748395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.769865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:38.770675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.771137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.771554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.772338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:38.774445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:38.856038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:38.857759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ade891d-62fd-4f4a-96b4-131cae39a886 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:38.895515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 852/1708] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:38 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 362 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:38.896929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:38.897214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:38.897485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:38.938828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:38.941843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.971772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 853/1709] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:38.973001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:38.976967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:38.996465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 857/1710] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:38 2022] POST /identity/v3/projects => generated 387 bytes in 463 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:38.998079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:39.000249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:39.139823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-393f3346-45a8-429d-ab6f-fce2247284d4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:39.156762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:39.157560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.159713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.161498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.170491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-913b00ad-7734-4880-a234-b82148289e91 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:39.184705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:39.185248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.185527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.185794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.337236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:39.339593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:39.342298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:39.351624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:39.354591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:39.369333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 854/1711] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 397 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:39.373462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:39.382061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 858/1712] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:38 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 384 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:39.388926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:39.388926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:39.399286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aa90088-9f54-4c25-89d7-3d9f51eb9224 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:39.409455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:39.416184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-777b23b9-253a-40ff-a5f0-a9aee35180e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:39.417115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777b23b9-253a-40ff-a5f0-a9aee35180e4 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.423205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777b23b9-253a-40ff-a5f0-a9aee35180e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.423205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-777b23b9-253a-40ff-a5f0-a9aee35180e4 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.423205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777b23b9-253a-40ff-a5f0-a9aee35180e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:39.423205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-777b23b9-253a-40ff-a5f0-a9aee35180e4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:39.533593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee874a1-93e0-4005-968e-ff5e7cdbf6b9 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:39.538729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:39.540036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.541046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.541499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.543653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:39.622931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:39.624882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:39.654419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 855/1713] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:39 2022] POST /identity/v3/users => generated 300 bytes in 279 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:39.654419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.654419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.654419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.661569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 859/1714] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:39 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 279 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:39.664627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:39.688729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baa473d0-6585-411a-a69d-878291f0da82 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:39.695601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:39.698283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.698711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.699134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.699996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:39.703129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:39.723636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 860/1715] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:39 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:39.726810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:39.742228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6904b830-8e7d-440c-8774-0dfbdbc280dd admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:39.750082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:39.752918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:39.752918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:39.754516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] PATH_INFO: `/v3/projects/05e831f5519243b1ab46c4439335b164/users/3021c175b7444e9f99f6073ab4acf2f0/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:39.810577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05e831f5519243b1ab46c4439335b164, user_id=3021c175b7444e9f99f6073ab4acf2f0, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:39.818621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:39.824433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] MFA Rules not processed for user `e8d3af2b82b641aea11ebf79677a3af1`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:39.826247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:39.956997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 861/1716] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:39 2022] PUT /identity/v3/projects/05e831f5519243b1ab46c4439335b164/users/3021c175b7444e9f99f6073ab4acf2f0/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:39.960647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:40.023719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 856/1717] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:39 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 372 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:40.058741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:40.150806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a8d3a7f-6a14-485c-83dd-f2c1f0d4ef31 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:40.164067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:40.164673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:40.164958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:40.165229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:40.167848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:40.168705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:40.185048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 862/1718] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:39 2022] GET /identity/v3/roles => generated 1420 bytes in 225 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:36:40.189078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:40.203497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8574c92e-7366-45a1-9c9e-3456b10f44a3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:40.209233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00a71efe-62d0-4e03-ac19-3fc9e280fcea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:40.214077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a71efe-62d0-4e03-ac19-3fc9e280fcea admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:40.214077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a71efe-62d0-4e03-ac19-3fc9e280fcea admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:40.214077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00a71efe-62d0-4e03-ac19-3fc9e280fcea admin admin] PATH_INFO: `/v3/projects/05e831f5519243b1ab46c4439335b164/users/3021c175b7444e9f99f6073ab4acf2f0/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:40.215375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a71efe-62d0-4e03-ac19-3fc9e280fcea admin admin] RBAC: Authorizing `identity:create_grant(project_id=05e831f5519243b1ab46c4439335b164, user_id=3021c175b7444e9f99f6073ab4acf2f0, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:40.217182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19b2eee-75f3-4b5f-9085-6449ddee61ef None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:40.222215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00a71efe-62d0-4e03-ac19-3fc9e280fcea admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:40.241137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 863/1719] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:40 2022] PUT /identity/v3/projects/05e831f5519243b1ab46c4439335b164/users/3021c175b7444e9f99f6073ab4acf2f0/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:40.252451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:40.253242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:40.253412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:40.301586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] MFA Rules not processed for user `3021c175b7444e9f99f6073ab4acf2f0`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:40.303007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:40.377415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:40.423177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:40.423845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:40.424467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:40.424467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:40.554690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:40.564679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 864/1720] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:40 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 314 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:40.668350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:40.669874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:40.677669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:40.679665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:40.694342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 857/1721] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:40 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 636 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:36:40.862574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9e0fbc7-c310-4518-976e-f3069b4557a6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:40.871904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:40.872558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:40.872833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:40.873107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:40.898710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:40.900852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:40.912833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:40.914470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:40.921024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 865/1722] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:40 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 253 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:36:40.971996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4adcf2-f325-4609-8288-ce00cf5864d5 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:41.053090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:41.194743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:41.194743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:41.194743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:41.194743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] PATH_INFO: `/v3/projects/36c1390c5c194a13b1801a9ef3cd1345` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:41.196384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=36c1390c5c194a13b1801a9ef3cd1345)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:41.198992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:41.233027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d58ad71-1568-4088-b252-a8b054debc15 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:41.243901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:41.244695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:41.246931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:41.247783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] PATH_INFO: `/v3/users/b53329615627482792c4817e98f118f2` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:41.250646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b53329615627482792c4817e98f118f2)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:41.256785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:41.542921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '36c1390c5c194a13b1801a9ef3cd1345', 'request_id': 'req-d88f5165-337a-42d9-869e-8d7374d390ef'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.545293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b53329615627482792c4817e98f118f2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.547392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b53329615627482792c4817e98f118f2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.568900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '36c1390c5c194a13b1801a9ef3cd1345', 'request_id': 'req-d88f5165-337a-42d9-869e-8d7374d390ef'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.574214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b53329615627482792c4817e98f118f2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.576908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b53329615627482792c4817e98f118f2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.637704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '36c1390c5c194a13b1801a9ef3cd1345', 'request_id': 'req-d88f5165-337a-42d9-869e-8d7374d390ef'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.655425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] The token cache is being invalidate because project 36c1390c5c194a13b1801a9ef3cd1345 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:41.655425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.656121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.676074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 858/1723] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:40 2022] DELETE /identity/v3/projects/36c1390c5c194a13b1801a9ef3cd1345 => generated 0 bytes in 761 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:41.676074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:41.679715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:41.718651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-58a8b5a7-abcc-47c4-8d68-eaa03b5f0163 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b53329615627482792c4817e98f118f2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:41.758743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 866/1724] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:41 2022] DELETE /identity/v3/users/b53329615627482792c4817e98f118f2 => generated 0 bytes in 703 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:41.762480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ffaaf1-8a73-4087-907e-3b1d363d669b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:41.763644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ffaaf1-8a73-4087-907e-3b1d363d669b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:41.764303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ffaaf1-8a73-4087-907e-3b1d363d669b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:41.895871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d88f5165-337a-42d9-869e-8d7374d390ef admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:41.922499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-35ffaaf1-8a73-4087-907e-3b1d363d669b None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:41.922499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-35ffaaf1-8a73-4087-907e-3b1d363d669b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.022135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:42.023631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.024189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.025415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] PATH_INFO: `/v3/projects/1707bf798d9f4731ba5a5aa8edb97d19` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.027035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1707bf798d9f4731ba5a5aa8edb97d19)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:42.029278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:42.084011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 867/1725] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:36:41 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:42.085400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf37dd1-fe6f-4c21-8f22-c5c866a34d4e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.085687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf37dd1-fe6f-4c21-8f22-c5c866a34d4e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.086096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf37dd1-fe6f-4c21-8f22-c5c866a34d4e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.116635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-acf37dd1-fe6f-4c21-8f22-c5c866a34d4e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:42.119785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-acf37dd1-fe6f-4c21-8f22-c5c866a34d4e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.154133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 868/1726] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 63 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:42.154133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c6ce5-afa6-48d2-980b-d825c1617ffa None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.154133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c6ce5-afa6-48d2-980b-d825c1617ffa None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.154133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-236c6ce5-afa6-48d2-980b-d825c1617ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.186953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-236c6ce5-afa6-48d2-980b-d825c1617ffa None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:42.191373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-236c6ce5-afa6-48d2-980b-d825c1617ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.204472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 869/1727] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:42.205938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68422a-3507-4d1b-a663-ee6a37486b14 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.206358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68422a-3507-4d1b-a663-ee6a37486b14 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.207146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb68422a-3507-4d1b-a663-ee6a37486b14 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.256239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1707bf798d9f4731ba5a5aa8edb97d19', 'request_id': 'req-5e511419-fdc2-413d-80c5-6824afd2c117'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:42.267646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cb68422a-3507-4d1b-a663-ee6a37486b14 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:42.268466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1707bf798d9f4731ba5a5aa8edb97d19', 'request_id': 'req-5e511419-fdc2-413d-80c5-6824afd2c117'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:42.268626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cb68422a-3507-4d1b-a663-ee6a37486b14 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.356964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1707bf798d9f4731ba5a5aa8edb97d19', 'request_id': 'req-5e511419-fdc2-413d-80c5-6824afd2c117'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:42.389890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] The token cache is being invalidate because project 1707bf798d9f4731ba5a5aa8edb97d19 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:42.391570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:42.397425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:42.397742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 859/1728] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:41 2022] DELETE /identity/v3/projects/1707bf798d9f4731ba5a5aa8edb97d19 => generated 0 bytes in 724 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:42.401011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:42.406840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.441517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 870/1729] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:42.447135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.447556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.447844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.523809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:42.527168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.543133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e511419-fdc2-413d-80c5-6824afd2c117 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:42.547812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:42.548990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.548990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.549489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.552030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.565270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 871/1730] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:42.566315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:42.581438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:42.590956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39196607-9938-4b5a-9982-58cb2b26035f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:42.608165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 860/1731] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:42 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:42.610802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.611120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.611392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.612678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87444c95-5e6e-4d6f-a5b2-b4186268d423 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:42.626410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:42.627759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.631562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.632084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.638547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:42.639220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:42.647358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:42.654568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:42.656998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.674924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:42.697384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 861/1732] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:42 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:42.698828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:42.699619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 872/1733] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:42.703671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:42.708446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.736796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-788d7160-37e5-4f6a-9480-7dc5e8dd7740 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:42.748856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:42.751224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.751507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.751778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.752711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:42.752711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:42.762595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:42.817061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 873/1734] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:42.824700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:42.862735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8abe2b3b-849b-422e-97d2-edf1dadb2281 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:42.887416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:42.900778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:42.908169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8368c00e-f3ec-4a7f-8e27-f007e1234a3c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:42.909165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.909311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.922953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:42.926901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:42.926901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:42.926901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.927488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:42.928321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:42.929837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:42.933892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.108371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.112998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.115273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 874/1735] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:42 2022] POST /identity/v3/projects => generated 369 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:43.117327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.126548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 862/1736] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:36:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 429 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:36:43.127525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.139390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a8df88-e2f2-4028-9595-48b59e1b8483 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:43.237682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:43.238452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.239044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.239418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.240228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.243608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.313092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ca135f-6416-4a4b-8c68-9ff582b2bc7f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:43.319581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:43.320217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.320501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.320832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.321548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.324232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.397272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 863/1737] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:43 2022] POST /identity/v3/users => generated 285 bytes in 267 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:43.399329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.421167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da25a963-0e7a-4e4b-bc6e-1c9eefcf674b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:43.436729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:43.437367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.437646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.437918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.438747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.440770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.461797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 875/1738] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:43 2022] POST /identity/v3/projects => generated 383 bytes in 345 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:43.468139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 864/1739] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:43 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:43.469458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.472198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.490255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3e9da6-d47d-4ff2-a23c-47c8a402fe59 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:43.608770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:43.693671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:43.694593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.694907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.695185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.698579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.700964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9599b555-ceb5-4c46-abc9-7e6dfe9714b7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:43.704336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.707677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:43.708552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.708999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.709490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.710348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.713600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.738690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 865/1740] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:43 2022] GET /identity/v3/roles => generated 1420 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:43.738690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.755324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00db214e-7fea-4027-ac93-a47ef71a0587 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:43.768254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc13e1d8-f1d6-4af9-9990-8fe3dd8cb109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:43.768954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13e1d8-f1d6-4af9-9990-8fe3dd8cb109 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.769291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13e1d8-f1d6-4af9-9990-8fe3dd8cb109 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.773806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc13e1d8-f1d6-4af9-9990-8fe3dd8cb109 admin admin] PATH_INFO: `/v3/projects/b19dd00581bf42449a63d13a2fdb27b0/users/ed1cd659a4c64a07a56b87f57660ae56/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.804121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc13e1d8-f1d6-4af9-9990-8fe3dd8cb109 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b19dd00581bf42449a63d13a2fdb27b0, user_id=ed1cd659a4c64a07a56b87f57660ae56, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:43.807972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc13e1d8-f1d6-4af9-9990-8fe3dd8cb109 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:43.870280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 876/1741] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:43 2022] POST /identity/v3/users => generated 292 bytes in 407 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:43.871953 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:43.894800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 866/1742] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:43 2022] PUT /identity/v3/projects/b19dd00581bf42449a63d13a2fdb27b0/users/ed1cd659a4c64a07a56b87f57660ae56/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:43.898982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:43.899324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:43.899719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:43.917373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8fe104-0f2b-4b4b-8948-f6bb166b20b7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:44.018990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] MFA Rules not processed for user `ed1cd659a4c64a07a56b87f57660ae56`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:44.020966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:44.066172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:44.066873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:44.067157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:44.067551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:44.068260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:44.071024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:44.098504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 877/1743] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:43 2022] GET /identity/v3/roles => generated 1420 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:44.101261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:44.129702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37a08945-67a0-4d98-a0fd-5fe271c31783 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:44.153720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:44.156361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:44.156848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:44.156848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:44.159430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:44.166129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:44.225567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 878/1744] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:44 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:44.246162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:44.264334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 867/1745] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:43 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 367 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:44.271661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db48e169-b2a6-47d0-ae2a-974bc5714064 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:44.283030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:44.302325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11ad18b0-6c2b-4c9d-8e63-327f2e476853 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:44.303123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ad18b0-6c2b-4c9d-8e63-327f2e476853 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:44.305574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ad18b0-6c2b-4c9d-8e63-327f2e476853 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:44.318722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d601662f-aec8-4e36-8e20-5c2b913fb8ff None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:44.319041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11ad18b0-6c2b-4c9d-8e63-327f2e476853 admin admin] PATH_INFO: `/v3/projects/cc44eab05e2a4639a54f43a3212dee90/users/5c3cefb475384e4e9936badbfb10295d/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:44.343857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f789617-2246-4bd8-a7a0-901f8e6a4bfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:44.346871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f789617-2246-4bd8-a7a0-901f8e6a4bfa admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:44.347220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f789617-2246-4bd8-a7a0-901f8e6a4bfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:44.347513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f789617-2246-4bd8-a7a0-901f8e6a4bfa admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:44.348437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f789617-2246-4bd8-a7a0-901f8e6a4bfa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:44.356978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f789617-2246-4bd8-a7a0-901f8e6a4bfa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:44.382167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ad18b0-6c2b-4c9d-8e63-327f2e476853 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc44eab05e2a4639a54f43a3212dee90, user_id=5c3cefb475384e4e9936badbfb10295d, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:44.400619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11ad18b0-6c2b-4c9d-8e63-327f2e476853 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:44.483555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 879/1746] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:44 2022] PUT /identity/v3/projects/cc44eab05e2a4639a54f43a3212dee90/users/5c3cefb475384e4e9936badbfb10295d/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:44.497804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:44.498721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:44.503782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:44.593579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 868/1747] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:44 2022] POST /identity/v3/projects => generated 369 bytes in 313 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:44.594606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] MFA Rules not processed for user `5c3cefb475384e4e9936badbfb10295d`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:44.596827 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:44.597115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:44.597453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:44.598730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:44.610705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:44.640111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] MFA Rules not processed for user `1dbfb1a3e99c418098c2b02aea93b923`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:44.641711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:44.821002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 880/1748] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:44 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 324 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:44.823144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:44.848170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 869/1749] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:44 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:44.851362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.073782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed063712-99a3-427c-a5f9-dccadad572db None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.089172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.090342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.090943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.091247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.092427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.094659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89782daf-72c4-48e3-86e9-4e897dfb0c9a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.096661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.107032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:45.127066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.131877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.131877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.131877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.131877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.131877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.180869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:45.232684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 881/1750] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:44 2022] POST /identity/v3/users => generated 291 bytes in 411 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:45.235942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.265544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ace74e1-d498-4a5e-82f3-f825b053a89e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.271648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.273016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.273556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.274054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.274628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 870/1751] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:44 2022] POST /identity/v3/projects => generated 385 bytes in 426 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:45.275890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.278157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.280573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.307133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 882/1752] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:45.308475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.327510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3a0e1b9-47a3-4289-a6b2-78a3d806ad63 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.500794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.501739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.503655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.504402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] PATH_INFO: `/v3/projects/a67c9515ecca4ed68247ed389a5870c6/users/7a666329e1a4469c93d889c102a15dc9/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.549645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a63d0e99-1446-45b4-a444-d8a40d15a980 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.554219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a67c9515ecca4ed68247ed389a5870c6, user_id=7a666329e1a4469c93d889c102a15dc9, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.566308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.572337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.579715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.580013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.580291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.581162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.599223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.664384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 883/1753] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:45 2022] PUT /identity/v3/projects/a67c9515ecca4ed68247ed389a5870c6/users/7a666329e1a4469c93d889c102a15dc9/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 356 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:45.671847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.754653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 871/1754] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:45 2022] POST /identity/v3/users => generated 293 bytes in 476 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:45.759009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.761847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:45.816570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b2bfe90-0c7f-486b-8e92-adba23c093c5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.869414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36eca2eb-2564-4f8b-b75a-53012664b38b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.876489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.879110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.879110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.879110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.879110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.880797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.905222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 884/1755] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 236 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:45.910335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:45.954883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c67e7a10-542f-42a9-960c-3168ed17f154 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:45.961514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bb79492-52c2-4472-80f3-7d53f581596f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:45.962948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb79492-52c2-4472-80f3-7d53f581596f admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.963515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb79492-52c2-4472-80f3-7d53f581596f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.964589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bb79492-52c2-4472-80f3-7d53f581596f admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:45.965588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb79492-52c2-4472-80f3-7d53f581596f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:45.967420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bb79492-52c2-4472-80f3-7d53f581596f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:45.991329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 885/1756] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:45 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:45.992920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:45.993340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:45.994184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.017939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:46.019409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:46.020114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:46.020575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] PATH_INFO: `/v3/projects/f777db7c495e4c70a575eb9ff6edcbe9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.021968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f777db7c495e4c70a575eb9ff6edcbe9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:46.035099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] MFA Rules not processed for user `28f0f9fef2f54e56a7a92d077c25d2b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:46.037554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:46.044618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:46.189961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f777db7c495e4c70a575eb9ff6edcbe9', 'request_id': 'req-bc7c3ed5-e812-4a71-a81b-9281d393d688'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.200645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 886/1757] 10.222.0.33 () {58 vars in 1015 bytes} [Tue Sep 27 16:36:45 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 209 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:46.202707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:46.213279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f777db7c495e4c70a575eb9ff6edcbe9', 'request_id': 'req-bc7c3ed5-e812-4a71-a81b-9281d393d688'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.217764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abbe5aff-cd06-441b-b367-8fe08589e142 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:46.303840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:46.305482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:46.306762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:46.307241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] PATH_INFO: `/v3/projects/a67c9515ecca4ed68247ed389a5870c6/users/7a666329e1a4469c93d889c102a15dc9/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.311541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a67c9515ecca4ed68247ed389a5870c6, user_id=7a666329e1a4469c93d889c102a15dc9, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:46.331041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:46.355904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f777db7c495e4c70a575eb9ff6edcbe9', 'request_id': 'req-bc7c3ed5-e812-4a71-a81b-9281d393d688'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.359991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 887/1758] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:46 2022] PUT /identity/v3/projects/a67c9515ecca4ed68247ed389a5870c6/users/7a666329e1a4469c93d889c102a15dc9/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:46.366377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:46.374085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] The token cache is being invalidate because project f777db7c495e4c70a575eb9ff6edcbe9 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:46.384975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:46.385077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.385077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.395098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 872/1759] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:45 2022] DELETE /identity/v3/projects/f777db7c495e4c70a575eb9ff6edcbe9 => generated 0 bytes in 628 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:46.396121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:46.400099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:46.435062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc7c3ed5-e812-4a71-a81b-9281d393d688 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:46.511558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:46.543626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99528cb1-f1f6-4e76-a589-a2c93504d3e5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:46.560731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:46.561823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:46.562277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:46.563818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] PATH_INFO: `/v3/users/b491da095fef45c792692be8cd08129b` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.567513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b491da095fef45c792692be8cd08129b)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:46.572754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:46.591496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:46.592455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:46.592949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:46.595047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.596176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:46.612902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:46.627298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 873/1760] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:46 2022] GET /identity/v3/roles => generated 1420 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:46.628469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:46.632268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:46.723355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fbde520-5ca2-431a-8be4-cb8656eba6b5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:46.734620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:46.744264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:46.744264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:46.744264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.744264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:46.763356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:46.766671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81250e5c-5b2c-4196-a2e2-c721f530bc5f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:46.780839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 874/1761] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:46 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:46.782004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:46.782286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:46.782555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:46.880886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b491da095fef45c792692be8cd08129b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.882211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b491da095fef45c792692be8cd08129b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.906304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] MFA Rules not processed for user `7a666329e1a4469c93d889c102a15dc9`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:46.908182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:46.931592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b491da095fef45c792692be8cd08129b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:46.932887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b491da095fef45c792692be8cd08129b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.002709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b491da095fef45c792692be8cd08129b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.037674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 888/1762] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:46 2022] DELETE /identity/v3/users/b491da095fef45c792692be8cd08129b => generated 0 bytes in 677 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:47.039238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:47.062021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:47.249353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 875/1763] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:46 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 450 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:47.251337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:47.254435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:47.310150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488fd57f-6b2f-459a-9dbf-1356cf921fb7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:47.314026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb65a3a0-f645-49b5-aacb-5fc86b26555e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:47.319270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6629081e-5101-4ccd-bc31-beee558e5db4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:47.320089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6629081e-5101-4ccd-bc31-beee558e5db4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:47.320355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6629081e-5101-4ccd-bc31-beee558e5db4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:47.320693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6629081e-5101-4ccd-bc31-beee558e5db4 admin admin] PATH_INFO: `/v3/projects/762b18ce44b845238df5e7a6036e5743/users/b89a4993963f48e19a3fea58cf864017/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:47.322314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:47.323035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:47.323821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:47.323821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] PATH_INFO: `/v3/users/3021c175b7444e9f99f6073ab4acf2f0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:47.329551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=3021c175b7444e9f99f6073ab4acf2f0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:47.339491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:47.369909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:47.392811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6629081e-5101-4ccd-bc31-beee558e5db4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=762b18ce44b845238df5e7a6036e5743, user_id=b89a4993963f48e19a3fea58cf864017, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:47.395775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6629081e-5101-4ccd-bc31-beee558e5db4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:47.512848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 876/1764] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:47 2022] PUT /identity/v3/projects/762b18ce44b845238df5e7a6036e5743/users/b89a4993963f48e19a3fea58cf864017/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:47.514689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47de0506-dece-480d-b713-51acf1018334 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:47.521972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47de0506-dece-480d-b713-51acf1018334 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:47.523294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47de0506-dece-480d-b713-51acf1018334 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:47.573284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-47de0506-dece-480d-b713-51acf1018334 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:47.578447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-47de0506-dece-480d-b713-51acf1018334 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:47.614205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3021c175b7444e9f99f6073ab4acf2f0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.615141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3021c175b7444e9f99f6073ab4acf2f0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.639518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3021c175b7444e9f99f6073ab4acf2f0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.642095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3021c175b7444e9f99f6073ab4acf2f0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.746366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3021c175b7444e9f99f6073ab4acf2f0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:47.777648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 889/1765] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:47 2022] DELETE /identity/v3/users/3021c175b7444e9f99f6073ab4acf2f0 => generated 0 bytes in 739 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:47.778891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:47.781397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:47.826650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 877/1766] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 313 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:47.833542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:47.833542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:47.833542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:47.934452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] MFA Rules not processed for user `b89a4993963f48e19a3fea58cf864017`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:47.934452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:47.964566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d683ff8d-672f-4575-b706-122a51c667f9 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:47.973208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:47.975864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:47.976666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:47.979247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:48.000790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:48.041247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 878/1767] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:47 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:48.042383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:48.044380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:48.108607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:48.124265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-896ad789-9085-4dad-8149-de34c6596ba2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:48.134552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:48.135880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:48.140507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:48.140507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:48.173981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:48.185634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:48.244923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 890/1768] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:47 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 466 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:48.246003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:48.253490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:48.302468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:48.312605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:48.377110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 879/1769] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:48 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:48.488766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c68ac07-05b2-457a-8d01-ab8c2bb84b23 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:48.492052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:48.497479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:48.498287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:48.498561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:48.498730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:48.499413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:48.501706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:48.514343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb94accd-5d73-4c51-9c34-cad4ec87289e service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:48.533924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:48.535004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:48.537077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:48.539530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:48.556944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:48.626942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:48.698520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 891/1770] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:48 2022] POST /identity/v3/projects => generated 383 bytes in 453 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:48.703048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:48.951351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:48.954663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:48.975234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 880/1771] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:48 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 485 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:48.976272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:48.984523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:49.052923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8862f198-0269-4afc-aeda-9b8b1af8ae3c service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:49.066584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba351dd9-4527-48c6-8d07-00e7d428ca9e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:49.073190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:49.075532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.075798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.078267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] PATH_INFO: `/v3/projects/05e831f5519243b1ab46c4439335b164` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.082849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=05e831f5519243b1ab46c4439335b164)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:49.089848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:49.091425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.091702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.095396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:49.103754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.103754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:49.107637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:49.284788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 892/1772] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:48 2022] POST /identity/v3/users => generated 298 bytes in 583 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:49.288253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:49.306864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16956373-c1ba-4227-b058-dcee545bc17e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:49.317837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b23db1-f5c6-402f-bbc3-bd1ac7bf8c16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:49.320086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b23db1-f5c6-402f-bbc3-bd1ac7bf8c16 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.320666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b23db1-f5c6-402f-bbc3-bd1ac7bf8c16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.333197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b23db1-f5c6-402f-bbc3-bd1ac7bf8c16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.336055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b23db1-f5c6-402f-bbc3-bd1ac7bf8c16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:49.341011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b23db1-f5c6-402f-bbc3-bd1ac7bf8c16 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:49.379431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05e831f5519243b1ab46c4439335b164', 'request_id': 'req-b168a6d2-7446-447f-96c6-498653e989f9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:49.391767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 893/1773] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:49 2022] GET /identity/v3/roles => generated 1420 bytes in 104 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:49.395291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.396972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.398042 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.404121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05e831f5519243b1ab46c4439335b164', 'request_id': 'req-b168a6d2-7446-447f-96c6-498653e989f9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:49.450053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] MFA Rules not processed for user `e8d3af2b82b641aea11ebf79677a3af1`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:49.454631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:49.544146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05e831f5519243b1ab46c4439335b164', 'request_id': 'req-b168a6d2-7446-447f-96c6-498653e989f9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:49.563870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] The token cache is being invalidate because project 05e831f5519243b1ab46c4439335b164 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:49.566547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:49.568432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:49.569276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:49.584413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 894/1775] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:49 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:49.584413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:49.584688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 881/1775] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:48 2022] DELETE /identity/v3/projects/05e831f5519243b1ab46c4439335b164 => generated 0 bytes in 597 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:49.589014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:49.591912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:49.691458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bfa4da2-32f9-4b20-8297-79422cbc6e45 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:49.710475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49fdcf4-052a-40f7-b281-a7eb9220a9b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:49.716443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49fdcf4-052a-40f7-b281-a7eb9220a9b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.716443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49fdcf4-052a-40f7-b281-a7eb9220a9b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.716443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49fdcf4-052a-40f7-b281-a7eb9220a9b1 admin admin] PATH_INFO: `/v3/projects/96955b3b25024a7493f2d5cd1357c814/users/e393b225194347f090862ba815750a90/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.751998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49fdcf4-052a-40f7-b281-a7eb9220a9b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96955b3b25024a7493f2d5cd1357c814, user_id=e393b225194347f090862ba815750a90, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:49.756269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49fdcf4-052a-40f7-b281-a7eb9220a9b1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:49.794956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b168a6d2-7446-447f-96c6-498653e989f9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:49.800895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:49.801562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.801839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.802116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] PATH_INFO: `/v3/projects/e98aef08432e4f3889f974b8a2b70f83` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.803522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] RBAC: Authorizing `identity:delete_project(project_id=e98aef08432e4f3889f974b8a2b70f83)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:49.808449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:49.830274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 895/1776] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:49 2022] PUT /identity/v3/projects/96955b3b25024a7493f2d5cd1357c814/users/e393b225194347f090862ba815750a90/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:49.831532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:49.833241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:49.833441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:49.876696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:49.878421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:49.971906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e98aef08432e4f3889f974b8a2b70f83', 'request_id': 'req-a50dbdf4-035b-42b4-b543-34db8027e0cd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:49.984280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e98aef08432e4f3889f974b8a2b70f83', 'request_id': 'req-a50dbdf4-035b-42b4-b543-34db8027e0cd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.000057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 896/1777] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:49 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:50.001795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:50.003334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.017058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3be2e809-6dfc-4f2d-a45e-2eaee7fcd5bd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:50.065715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e98aef08432e4f3889f974b8a2b70f83', 'request_id': 'req-a50dbdf4-035b-42b4-b543-34db8027e0cd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.084592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] The token cache is being invalidate because project e98aef08432e4f3889f974b8a2b70f83 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:50.085122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.087117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a50dbdf4-035b-42b4-b543-34db8027e0cd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.090943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 882/1778] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:49 2022] DELETE /identity/v3/projects/e98aef08432e4f3889f974b8a2b70f83 => generated 0 bytes in 508 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:50.093581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.094311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.094727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.167169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:50.168119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.168854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.168854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.169289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:50.171008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:50.175674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:50.177950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.191950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 897/1779] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:50 2022] GET /identity/v3/roles => generated 1420 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:50.193677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:50.208354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 883/1780] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:50 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:50.209613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:50.216043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.220155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.265491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20944635-ac1e-436a-aed2-5ed79dcfa025 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:50.265491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:50.265491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.265491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.266028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] PATH_INFO: `/v3/users/baa6149227c842a1907adb8950ed1a2e` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.266028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] RBAC: Authorizing `identity:delete_user(user_id=baa6149227c842a1907adb8950ed1a2e)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:50.266028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:50.289153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:50.302942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57d85923-0b0e-4c77-9f53-7233f305ce3a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:50.336365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:50.338184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.347734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.348168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] PATH_INFO: `/v3/projects/96955b3b25024a7493f2d5cd1357c814/users/e393b225194347f090862ba815750a90/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.350299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96955b3b25024a7493f2d5cd1357c814, user_id=e393b225194347f090862ba815750a90, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:50.354571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:50.393710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 884/1781] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:50 2022] PUT /identity/v3/projects/96955b3b25024a7493f2d5cd1357c814/users/e393b225194347f090862ba815750a90/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:50.393710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:50.542882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'baa6149227c842a1907adb8950ed1a2e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.553521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'baa6149227c842a1907adb8950ed1a2e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.564118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'baa6149227c842a1907adb8950ed1a2e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.567874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'baa6149227c842a1907adb8950ed1a2e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.573691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aeefd60a-aa28-4284-96f1-c519c11b8365 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:50.589687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:50.591516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.603181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.603783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.609786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:50.619043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:50.624543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3eae75c4-8587-4c4c-b8ee-519112b2d91d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:50.640524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 885/1782] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:50.642304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.642596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.642960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.700584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'baa6149227c842a1907adb8950ed1a2e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:50.708235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] MFA Rules not processed for user `e393b225194347f090862ba815750a90`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:50.709674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.750126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 898/1783] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:50 2022] DELETE /identity/v3/users/baa6149227c842a1907adb8950ed1a2e => generated 0 bytes in 557 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:50.751234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:50.752817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.854712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 886/1784] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:50 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:50.862645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1ec090-b387-46a0-a82f-14ad9806752c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:50.868063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:50.869239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:50.869640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:50.869811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:50.870308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:50.870962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:50.873057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:50.873716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:50.995464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 899/1785] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:50 2022] POST /identity/v3/projects => generated 381 bytes in 244 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:50.999807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:51.025725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:51.027871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dfe5615-2bf2-4c53-8b82-2cf7cb268764 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:51.131982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:51.132618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.132895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.138257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.140986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:51.201874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c0cf8a6-6ae7-495e-9e01-609cd8829506 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:51.210272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:51.211087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.211649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.212826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.213652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:51.217464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:51.334423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:51.336534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:51.353946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 900/1786] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:50 2022] POST /identity/v3/users => generated 291 bytes in 355 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:51.355619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 887/1787] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:50 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:51.357634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:51.377065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-298715bb-4859-4c2e-97a7-64daf2eb229d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:51.391314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:51.391943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.392226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.392533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.393625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:51.394814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:51.423283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:51.432086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 901/1788] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 74 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:51.438832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1427761-b589-4103-b54d-2d33de41909c service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:51.454219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77327446-2716-44e5-a730-4a2c03ec5cea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:51.454219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77327446-2716-44e5-a730-4a2c03ec5cea admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.454927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77327446-2716-44e5-a730-4a2c03ec5cea admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.454927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77327446-2716-44e5-a730-4a2c03ec5cea admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.455517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77327446-2716-44e5-a730-4a2c03ec5cea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:51.458089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77327446-2716-44e5-a730-4a2c03ec5cea admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:51.500100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 888/1789] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:51 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:51.500644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:51.519310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd717aaf-6bde-43b4-ada7-401a340c6459 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:51.526208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c219aa4d-466b-45c8-b4c8-72efe85f4503 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:51.527810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219aa4d-466b-45c8-b4c8-72efe85f4503 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.529112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219aa4d-466b-45c8-b4c8-72efe85f4503 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.540471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c219aa4d-466b-45c8-b4c8-72efe85f4503 admin admin] PATH_INFO: `/v3/projects/8b28c20d84ed481f9f7a3a088ced3bbf/users/7b9f333ccb994483b0049ba3ea515209/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.605295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fb7a89-2538-49ee-a340-81cb196fc01b None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.605295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fb7a89-2538-49ee-a340-81cb196fc01b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.605295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fb7a89-2538-49ee-a340-81cb196fc01b None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.605295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 889/1790] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:36:51 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:36:51.605295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fb7a89-2538-49ee-a340-81cb196fc01b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:51.605295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03fb7a89-2538-49ee-a340-81cb196fc01b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:51.605778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c219aa4d-466b-45c8-b4c8-72efe85f4503 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8b28c20d84ed481f9f7a3a088ced3bbf, user_id=7b9f333ccb994483b0049ba3ea515209, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:51.605778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c219aa4d-466b-45c8-b4c8-72efe85f4503 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:51.639710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fb7a89-2538-49ee-a340-81cb196fc01b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:51.644691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:51.645435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.645959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.646398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] PATH_INFO: `/v3/projects/3cfe269f5cb749959937adef4c17915a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.647652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cfe269f5cb749959937adef4c17915a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:51.649618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:51.711949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 902/1791] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:51 2022] PUT /identity/v3/projects/8b28c20d84ed481f9f7a3a088ced3bbf/users/7b9f333ccb994483b0049ba3ea515209/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 205 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:51.713452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc36abb-bb69-4510-9347-475bdc383626 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.713741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc36abb-bb69-4510-9347-475bdc383626 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.714016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcc36abb-bb69-4510-9347-475bdc383626 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.735727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fcc36abb-bb69-4510-9347-475bdc383626 None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:51.736961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fcc36abb-bb69-4510-9347-475bdc383626 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:51.868792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cfe269f5cb749959937adef4c17915a', 'request_id': 'req-b8a56e8f-a9a6-40c2-9e04-e1753992006b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:51.876381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cfe269f5cb749959937adef4c17915a', 'request_id': 'req-b8a56e8f-a9a6-40c2-9e04-e1753992006b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:51.907411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:51.926212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 903/1792] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:36:51 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 213 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:51.927673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:51.927960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:51.928235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:51.989313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] MFA Rules not processed for user `7b9f333ccb994483b0049ba3ea515209`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:51.992042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cfe269f5cb749959937adef4c17915a', 'request_id': 'req-b8a56e8f-a9a6-40c2-9e04-e1753992006b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.009319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:52.018736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] The token cache is being invalidate because project 3cfe269f5cb749959937adef4c17915a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:52.020137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.025721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.031922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 890/1793] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:51 2022] DELETE /identity/v3/projects/3cfe269f5cb749959937adef4c17915a => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:52.036367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:52.038682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:52.168097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8a56e8f-a9a6-40c2-9e04-e1753992006b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:52.182750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:52.183855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:52.184316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:52.184728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] PATH_INFO: `/v3/users/55bfd6d2d0a14bdc909d5c407ef56c3b` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:52.186057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] RBAC: Authorizing `identity:delete_user(user_id=55bfd6d2d0a14bdc909d5c407ef56c3b)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:52.194996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:52.215379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 904/1794] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:51 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 288 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:52.219583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:52.223748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:52.313968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55a32203-30f7-4be5-9f03-8a9ed4a3c4fd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:52.353774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:52.355171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:52.357742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:52.358067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:52.360190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:52.375485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:52.377105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:52.390852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 905/1795] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:52 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 175 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:52.391717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:52.393407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:52.409496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8e16fc3-dd4b-4e1a-98c4-364e414a0553 service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:52.417272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:52.418419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:52.419068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:52.419364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:52.420016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:52.422840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:52.449965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '55bfd6d2d0a14bdc909d5c407ef56c3b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.458064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '55bfd6d2d0a14bdc909d5c407ef56c3b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.458951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '55bfd6d2d0a14bdc909d5c407ef56c3b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.475304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '55bfd6d2d0a14bdc909d5c407ef56c3b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.535343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:52.557546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '55bfd6d2d0a14bdc909d5c407ef56c3b'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:52.571775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 906/1796] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:52 2022] POST /identity/v3/projects => generated 381 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:52.576981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:52.581177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:52.594696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 891/1797] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:52 2022] DELETE /identity/v3/users/55bfd6d2d0a14bdc909d5c407ef56c3b => generated 0 bytes in 559 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:52.776224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53cd1b30-b91d-4339-a247-ddb2af1d24e3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:52.954068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:52.955021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:52.955358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:52.955639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:52.956309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:52.961136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:53.112656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 907/1798] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:52 2022] POST /identity/v3/users => generated 297 bytes in 537 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:53.118427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:53.135575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7576b866-22bc-484b-83b7-5d4a2cfd8228 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:53.156537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:53.156537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:53.156853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:53.157279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:53.158327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:53.166272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:53.190285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 892/1799] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 73 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:53.194354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:53.240261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-233c2376-257b-4cc0-bfaa-688283d75d5d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:53.254387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:53.255033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:53.255314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:53.256276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] PATH_INFO: `/v3/projects/1222eb00196740459dd73461e1dcb9b6/users/9834860f5d464a5f99fc5ad8dcca2c06/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:53.281681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] RBAC: Authorizing `identity:create_grant(project_id=1222eb00196740459dd73461e1dcb9b6, user_id=9834860f5d464a5f99fc5ad8dcca2c06, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:53.284825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:53.433930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 908/1800] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:53 2022] PUT /identity/v3/projects/1222eb00196740459dd73461e1dcb9b6/users/9834860f5d464a5f99fc5ad8dcca2c06/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 240 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:53.437717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:53.451551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae0e87f-9717-4361-af9b-e551d75d80bc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:53.568261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52caffbe-486c-4372-97f2-55ffc6c8b649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:53.568994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52caffbe-486c-4372-97f2-55ffc6c8b649 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:53.569343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52caffbe-486c-4372-97f2-55ffc6c8b649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:53.569749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52caffbe-486c-4372-97f2-55ffc6c8b649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:53.570529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52caffbe-486c-4372-97f2-55ffc6c8b649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:53.572724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52caffbe-486c-4372-97f2-55ffc6c8b649 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:53.586819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 893/1801] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:53 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:53.590610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:53.616022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb22236f-8ab3-49a5-bdce-e938c1f30bea admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:53.623671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:53.624753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:53.624753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:53.625029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] PATH_INFO: `/v3/projects/1222eb00196740459dd73461e1dcb9b6/users/9834860f5d464a5f99fc5ad8dcca2c06/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:53.627927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1222eb00196740459dd73461e1dcb9b6, user_id=9834860f5d464a5f99fc5ad8dcca2c06, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:53.633606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:53.666836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 909/1802] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:53 2022] PUT /identity/v3/projects/1222eb00196740459dd73461e1dcb9b6/users/9834860f5d464a5f99fc5ad8dcca2c06/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:53.681448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:53.682095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:53.682567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:53.733908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] MFA Rules not processed for user `9834860f5d464a5f99fc5ad8dcca2c06`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:53.735247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:53.887261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 894/1803] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:53 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 208 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:54.067971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:54.069758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:54.160196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2149b8d-91c4-4f18-9c4a-e8290203b2e2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:54.167996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:54.169111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:54.169569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:54.170181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:54.187027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:54.188662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:54.195558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:54.199022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:54.220972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 910/1804] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:54 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:54.236620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=82,keystone:DELETE=17,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:54.306453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:54.466176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d412d9e-22c5-474f-8fa3-476d1f97897b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:54.480203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:54.481341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:54.482432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:54.482940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] PATH_INFO: `/v3/projects/08d0c840081b419ab17893b5520a7bde` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:54.488524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] RBAC: Authorizing `identity:delete_project(project_id=08d0c840081b419ab17893b5520a7bde)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:54.491423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:54.654186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '08d0c840081b419ab17893b5520a7bde', 'request_id': 'req-b2d805bf-c8a0-41d3-9d98-9108e69e2075'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:54.666347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '08d0c840081b419ab17893b5520a7bde', 'request_id': 'req-b2d805bf-c8a0-41d3-9d98-9108e69e2075'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:54.685340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:54.688241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:54.700451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7092d5c-0a83-4e27-bf52-66642713261d service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:54.775933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '08d0c840081b419ab17893b5520a7bde', 'request_id': 'req-b2d805bf-c8a0-41d3-9d98-9108e69e2075'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:54.799707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:54.800466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:54.800898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:54.802329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:54.803270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] The token cache is being invalidate because project 08d0c840081b419ab17893b5520a7bde was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:54.803752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:54.806429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b2d805bf-c8a0-41d3-9d98-9108e69e2075 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:54.809710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:54.811310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 895/1805] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:54 2022] DELETE /identity/v3/projects/08d0c840081b419ab17893b5520a7bde => generated 0 bytes in 625 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:54.819513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:54.819792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:54.821762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:54.897103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:54.900307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:54.939993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:54.944031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-628c88a8-0f3a-4ec1-97fc-38939ec4c1b1 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:54.965279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 911/1806] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:54 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 280 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:36:55.081742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.082760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.083284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.087938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 896/1807] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 269 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:55.117635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:55.127984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:55.128636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 912/1808] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:55.133625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:55.164240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c772dbd9-99a8-447f-85a2-a98ba0516984 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:55.175670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:55.176631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.177040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.178709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.180115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:55.181376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:55.183111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:55.201778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 897/1809] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 69 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:55.205900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:55.216739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b307090-629c-4180-a558-c5677b6d1e3f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:55.229493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:55.231349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.231349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.231349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.232452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:55.234738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:55.402127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 913/1810] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:55 2022] POST /identity/v3/projects => generated 383 bytes in 196 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:55.405367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:55.422897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6e591-5a9b-44a4-8e3d-dc495796c6de admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:55.564671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:55.565492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.565787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.573774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.574787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:55.583180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:55.761484 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 898/1811] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:55 2022] POST /identity/v3/users => generated 292 bytes in 357 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:55.767528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:55.787027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b58c034-d402-4391-bad2-dbed4c6a8fe7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:55.796408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:55.797243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.797407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.798484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.799578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:55.802355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:55.822271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 914/1812] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:55 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:55.825983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:55.842970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fdaf5a7-2851-4459-befb-1d18011a031c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:55.851505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013652c5-6236-4eda-84c8-6b35c3ab0439 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:55.858170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013652c5-6236-4eda-84c8-6b35c3ab0439 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.858170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013652c5-6236-4eda-84c8-6b35c3ab0439 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.858170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013652c5-6236-4eda-84c8-6b35c3ab0439 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.858170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013652c5-6236-4eda-84c8-6b35c3ab0439 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:55.858170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013652c5-6236-4eda-84c8-6b35c3ab0439 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:55.878526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 899/1813] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:55 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:55.915243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:55.922191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f0e89d-fee6-404f-b2d1-6706884a34b3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:55.953900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:55.953900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:55.953900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:55.953900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] PATH_INFO: `/v3/projects/253f3ef5ee774e92b68232ac728f7f76/users/285fda448d3644ebb193a92eddfc892e/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:55.955328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] RBAC: Authorizing `identity:create_grant(project_id=253f3ef5ee774e92b68232ac728f7f76, user_id=285fda448d3644ebb193a92eddfc892e, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:55.956539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:56.872649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 915/1814] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:55 2022] PUT /identity/v3/projects/253f3ef5ee774e92b68232ac728f7f76/users/285fda448d3644ebb193a92eddfc892e/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 972 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:56.932496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:56.932496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:56.932496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:56.932496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] MFA Rules not processed for user `285fda448d3644ebb193a92eddfc892e`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:56.932496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:56.986727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 900/1815] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:56 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:36:56.992027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:57.079588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d69a77d-97ec-4545-8214-5380bf9af31c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:57.084743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:57.085565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:57.085844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:57.086115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:57.086751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:57.088783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:57.602429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 916/1816] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:56 2022] POST /identity/v3/projects => generated 383 bytes in 609 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:57.605519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:57.616980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be602593-6839-4f7b-a084-a3bc7f181f9e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:57.662863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:57.683058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:57.683696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:57.683989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:57.684262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:57.684928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:57.687034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:57.819231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 901/1817] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:36:57 2022] POST /identity/v3/users => generated 298 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:36:57.823897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:57.834383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9a1eaea-c0fe-4ff5-825d-3951350ca698 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:57.841189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:57.843979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:57.843979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:57.843979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:57.843979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:57.844983 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:57.848850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:57.853226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:57.857404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 917/1818] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:57 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:57.861738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:57.873802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df7abece-1a8e-4957-8d1f-1bdddd69b46f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:57.880204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47617b35-5168-4691-864e-700af7a6d37b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:57.880881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47617b35-5168-4691-864e-700af7a6d37b admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:57.881160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47617b35-5168-4691-864e-700af7a6d37b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:57.881438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47617b35-5168-4691-864e-700af7a6d37b admin admin] PATH_INFO: `/v3/projects/a50029e7d2984db5940577455e88f534/users/a48af238f391411e8d08acce64fa92e6/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:57.909665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47617b35-5168-4691-864e-700af7a6d37b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a50029e7d2984db5940577455e88f534, user_id=a48af238f391411e8d08acce64fa92e6, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:57.913311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47617b35-5168-4691-864e-700af7a6d37b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:57.958294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcb61cfc-2d55-4d00-8f64-2aec09008ee8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:57.965658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9834860f5d464a5f99fc5ad8dcca2c06', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1222eb00196740459dd73461e1dcb9b6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:57.966685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:57.967069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:57.967417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:57.968077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:57.968415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:57.972943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:57.986226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 902/1819] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:57.990564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.001098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e535a9b6-0f9f-40bc-9054-4e98c08a2f8c tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.008845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9834860f5d464a5f99fc5ad8dcca2c06', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1222eb00196740459dd73461e1dcb9b6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.009455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.009736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.010550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.011215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.013174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.028009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 918/1820] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:57 2022] PUT /identity/v3/projects/a50029e7d2984db5940577455e88f534/users/a48af238f391411e8d08acce64fa92e6/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:36:58.033575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f18326-e57b-4043-b471-2e210810520e None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.033575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f18326-e57b-4043-b471-2e210810520e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.033575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f18326-e57b-4043-b471-2e210810520e None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.043659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 919/1821] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:36:58 2022] GET /identity => generated 270 bytes in 15 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:36:58.044564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f18326-e57b-4043-b471-2e210810520e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.047063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b8f18326-e57b-4043-b471-2e210810520e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.106853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 903/1822] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:36:57 2022] POST /identity/v3/projects => generated 354 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:36:58.106853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.130881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f18326-e57b-4043-b471-2e210810520e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.191055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98031cd-9d77-4633-89b0-c806c9e9f1fe tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.222691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2cb586-6b54-41f3-a219-851ad3dae3f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.222691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2cb586-6b54-41f3-a219-851ad3dae3f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.222691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2cb586-6b54-41f3-a219-851ad3dae3f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.222691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2cb586-6b54-41f3-a219-851ad3dae3f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.223041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.223145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2cb586-6b54-41f3-a219-851ad3dae3f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.223236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.223236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.223236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] PATH_INFO: `/v3/users/6a153563a2c040de84179a4a2b456030` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.223473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2cb586-6b54-41f3-a219-851ad3dae3f2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.223562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] RBAC: Authorizing `identity:delete_user(user_id=6a153563a2c040de84179a4a2b456030)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.223562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.224612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 904/1823] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:36:58 2022] GET /identity/v3/roles => generated 1420 bytes in 122 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:36:58.225940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.226228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.226889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.250497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:58.252174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.271539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:58.348086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:36:58.352759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 905/1824] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:36:58 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:58.353626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.419418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-950fa952-8468-4bad-ae33-3d34e8e9c325 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.424042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9834860f5d464a5f99fc5ad8dcca2c06', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1222eb00196740459dd73461e1dcb9b6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.431681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.431681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.431681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.431681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:36:58.431681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.432792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.439334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 906/1825] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:36:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:36:58.440109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.450108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d0500-760e-49af-b5cc-131b0b9d0015 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.456076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.456371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.457032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.457032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] PATH_INFO: `/v3/projects/a50029e7d2984db5940577455e88f534/users/a48af238f391411e8d08acce64fa92e6/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.459108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a50029e7d2984db5940577455e88f534, user_id=a48af238f391411e8d08acce64fa92e6, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.461927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.464986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6a153563a2c040de84179a4a2b456030'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:58.466255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6a153563a2c040de84179a4a2b456030'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:58.481407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 907/1826] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:36:58 2022] PUT /identity/v3/projects/a50029e7d2984db5940577455e88f534/users/a48af238f391411e8d08acce64fa92e6/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:58.483367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.484296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.492590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6a153563a2c040de84179a4a2b456030'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:58.493388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6a153563a2c040de84179a4a2b456030'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:58.556783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6a153563a2c040de84179a4a2b456030'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:58.563393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87e2e1c-eae3-42ed-9551-7aa4dea06cf5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.573040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.575520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.575520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.575520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.576070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.581880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 920/1827] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:58 2022] DELETE /identity/v3/users/6a153563a2c040de84179a4a2b456030 => generated 0 bytes in 538 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:58.582908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.584237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.589746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.591521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f67f041-b1af-4b60-a796-624f479e96f1 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.604646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 908/1828] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:58 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:58.605978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140e4071-54c7-4509-8578-090bffd7343c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.606263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140e4071-54c7-4509-8578-090bffd7343c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.606537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-140e4071-54c7-4509-8578-090bffd7343c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.639577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-140e4071-54c7-4509-8578-090bffd7343c None None] MFA Rules not processed for user `a48af238f391411e8d08acce64fa92e6`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:58.640908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-140e4071-54c7-4509-8578-090bffd7343c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.667075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c169e9-c616-4439-b7f6-6b62e01b3bdd admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.667075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.667075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.667075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.667766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.670550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.740775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 909/1829] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:58 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:58.742116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.742408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.742842 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.753532 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.755049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.764606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:58.765544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 921/1830] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:58 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 183 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:58.765812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.768447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.776565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.836018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 910/1831] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:58.846573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987cd346-1525-43f9-898e-2dd021de3a67 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.846573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9834860f5d464a5f99fc5ad8dcca2c06', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '1222eb00196740459dd73461e1dcb9b6', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.846573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.846934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.846934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] PATH_INFO: `/v3/projects/0323a90e94af4a6b84fa819287371d60` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.846934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorizing `identity:delete_project(project_id=0323a90e94af4a6b84fa819287371d60)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.847132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.848039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.848344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.905948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c373f0-767e-4b5c-b2ef-284c1ac3cf2a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.909424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.910024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.910300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.910567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.920189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.921579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.930747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 911/1832] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:36:58.931544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:58.933527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:58.944556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603e584-0a70-47e6-9660-aaca4e3f69f3 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:58.948197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:58.948787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:58.949062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:58.949330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:58.958552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:58.960014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:58.970441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 912/1833] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:58 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:59.000115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0323a90e94af4a6b84fa819287371d60', 'request_id': 'req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.011423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:59.013461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:59.033252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ab8e9a7-a131-4954-a1d5-fb5f1eacf4a8 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:59.087070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0323a90e94af4a6b84fa819287371d60', 'request_id': 'req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.095225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0323a90e94af4a6b84fa819287371d60', 'request_id': 'req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.097706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:59.098754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:59.098754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:59.098954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] PATH_INFO: `/v3/users/ed1cd659a4c64a07a56b87f57660ae56` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:59.100267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ed1cd659a4c64a07a56b87f57660ae56)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:59.102550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:59.113054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] The token cache is being invalidate because project 0323a90e94af4a6b84fa819287371d60 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:36:59.114667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.115396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.116975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 922/1834] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:58 2022] DELETE /identity/v3/projects/0323a90e94af4a6b84fa819287371d60 => generated 0 bytes in 349 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:59.204281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:59.205825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:59.311386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db30d3b8-8b9e-47dd-b777-e9d70d79ce74 tempest-VolumeQuotaClassesTest-1601327053 tempest-VolumeQuotaClassesTest-1601327053-project-admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:59.317433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:59.319733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:59.321112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:59.321112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:59.323616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:59.356066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ed1cd659a4c64a07a56b87f57660ae56'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.363865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ed1cd659a4c64a07a56b87f57660ae56'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.364705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ed1cd659a4c64a07a56b87f57660ae56'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.365622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ed1cd659a4c64a07a56b87f57660ae56'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.418150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:59.419574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-558365cf-08cf-4c7e-aa4b-cd84180852bc service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:59.434836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 923/1835] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:59 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 232 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:59.443066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ed1cd659a4c64a07a56b87f57660ae56'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:36:59.476549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 913/1836] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:36:59 2022] DELETE /identity/v3/users/ed1cd659a4c64a07a56b87f57660ae56 => generated 0 bytes in 466 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:36:59.481503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:59.481787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:59.482593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:59.517712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:36:59.519461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:59.616832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 924/1837] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:36:59 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:36:59.647839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:59.654858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:59.780620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0ce37e1-ba78-4256-8e5b-4fe8d99e4b90 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:59.781784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:36:59.783119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:36:59.785808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:59.786884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:59.795268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:59.795268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:59.803454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:59.803633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:36:59.806483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-895edee9-abbd-4960-98cc-6d11987453ac None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:36:59.820175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 914/1838] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:36:59 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 173 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:36:59.824010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:36:59.824010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:36:59.824010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:36:59.824438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] PATH_INFO: `/v3/projects/44114768a46c440ebe3222da40801a9f` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:36:59.825696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] RBAC: Authorizing `identity:delete_project(project_id=44114768a46c440ebe3222da40801a9f)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:36:59.827854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:00.062803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '44114768a46c440ebe3222da40801a9f', 'request_id': 'req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.103588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:00.105855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:00.128564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17b59266-6717-4f81-8200-634323173cc8 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:00.163039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '44114768a46c440ebe3222da40801a9f', 'request_id': 'req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.174904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '44114768a46c440ebe3222da40801a9f', 'request_id': 'req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.187930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] The token cache is being invalidate because project 44114768a46c440ebe3222da40801a9f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:00.188224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.188919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.190905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 925/1839] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:36:59 2022] DELETE /identity/v3/projects/44114768a46c440ebe3222da40801a9f => generated 0 bytes in 410 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:00.195470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:00.197135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:00.237436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:00.238937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:00.240560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:00.240908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] PATH_INFO: `/v3/users/7b9f333ccb994483b0049ba3ea515209` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:00.243179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b9f333ccb994483b0049ba3ea515209)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:00.247008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:00.352989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e2b5cf-2ff9-4d1d-8ec3-cef610d72243 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:00.358811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:00.359963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:00.360529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:00.361111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] PATH_INFO: `/v3/users/98495404c1dd4416b9278fedfc9c1ae4` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:00.362277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=98495404c1dd4416b9278fedfc9c1ae4)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:00.364775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:00.492217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b9f333ccb994483b0049ba3ea515209'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.504958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b9f333ccb994483b0049ba3ea515209'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.506166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b9f333ccb994483b0049ba3ea515209'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.506833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b9f333ccb994483b0049ba3ea515209'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.590759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b9f333ccb994483b0049ba3ea515209'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.616151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 915/1840] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:00 2022] DELETE /identity/v3/users/7b9f333ccb994483b0049ba3ea515209 => generated 0 bytes in 514 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:00.618216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '98495404c1dd4416b9278fedfc9c1ae4'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.619743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:00.621403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '98495404c1dd4416b9278fedfc9c1ae4'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.621720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:00.641844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '98495404c1dd4416b9278fedfc9c1ae4'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.645153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '98495404c1dd4416b9278fedfc9c1ae4'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.728276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '98495404c1dd4416b9278fedfc9c1ae4'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:00.738723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c1d6ff2-54dd-4685-8de0-7de820f48651 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:00.746849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:00.747767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:00.748362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:00.748831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] PATH_INFO: `/v3/users/285fda448d3644ebb193a92eddfc892e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:00.751262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] RBAC: Authorizing `identity:delete_user(user_id=285fda448d3644ebb193a92eddfc892e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:00.753808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:00.783196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 926/1841] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:00 2022] DELETE /identity/v3/users/98495404c1dd4416b9278fedfc9c1ae4 => generated 0 bytes in 589 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:00.784684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:00.792667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:00.917929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc3e6ce7-7586-4cd0-b3c6-04391f0067e6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:00.923448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:00.924427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:00.924872 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:00.925481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] PATH_INFO: `/v3/projects/b19dd00581bf42449a63d13a2fdb27b0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:00.926509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b19dd00581bf42449a63d13a2fdb27b0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:00.928452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:01.039557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '285fda448d3644ebb193a92eddfc892e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.062637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '285fda448d3644ebb193a92eddfc892e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.065576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '285fda448d3644ebb193a92eddfc892e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.067085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '285fda448d3644ebb193a92eddfc892e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.075882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b19dd00581bf42449a63d13a2fdb27b0', 'request_id': 'req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.162074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2c6d1ea3-ad64-4bde-90b5-7a777c76e229 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '285fda448d3644ebb193a92eddfc892e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.196271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 916/1842] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:00 2022] DELETE /identity/v3/users/285fda448d3644ebb193a92eddfc892e => generated 0 bytes in 579 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:01.211542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.211763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.212048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.235028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:01.236265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:01.259804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b19dd00581bf42449a63d13a2fdb27b0', 'request_id': 'req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.271162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b19dd00581bf42449a63d13a2fdb27b0', 'request_id': 'req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.293135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] The token cache is being invalidate because project b19dd00581bf42449a63d13a2fdb27b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:01.293433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.294370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0b7aaa46-a2f8-423f-90de-02b28da2f0d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.296487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 927/1843] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:00 2022] DELETE /identity/v3/projects/b19dd00581bf42449a63d13a2fdb27b0 => generated 0 bytes in 513 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:01.297783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.298067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.298343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.310326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 917/1844] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:01.311324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:01.312718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:01.320596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:01.321795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:01.330552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d2edc2-d8fe-412f-a49f-5cbddd9f6e45 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:01.335377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:01.336155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.338699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.338699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] PATH_INFO: `/v3/users/7a666329e1a4469c93d889c102a15dc9` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.338929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 928/1845] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:01.345156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a666329e1a4469c93d889c102a15dc9)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:01.347964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:01.348088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:01.352088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:01.449160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f0e7399-9bc3-47ca-b942-21b22012d32b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:01.454683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:01.455364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.455644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.455912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.469409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:01.472298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:01.487106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 929/1846] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:01.487962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:01.497383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c628e26-d783-42ea-814d-d3df4cdf3d4c service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:01.506058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:01.508908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.508908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.508908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.519609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:01.521582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:01.528270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:01.535977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 930/1847] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:01 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:01.538014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:01.538878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:01.560841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7cb7027-5f92-461c-8ca6-35b57c807a6b service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:01.566595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:01.567501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.567918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.570182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] PATH_INFO: `/v3/projects/c41d2fe78f3c4f3cbb71565656e3548c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.570182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c41d2fe78f3c4f3cbb71565656e3548c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:01.573043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:01.621591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=4,keystone:SELECT=94,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:01.635980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a666329e1a4469c93d889c102a15dc9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.670540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a666329e1a4469c93d889c102a15dc9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.672336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a666329e1a4469c93d889c102a15dc9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.674057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a666329e1a4469c93d889c102a15dc9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.765066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c41d2fe78f3c4f3cbb71565656e3548c', 'request_id': 'req-b3282fad-e47e-49ad-88c2-19991500aa18'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.791314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e3bd1d07-44f8-4bb3-97a7-254e5638bde3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a666329e1a4469c93d889c102a15dc9'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.826824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 918/1848] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:01 2022] DELETE /identity/v3/users/7a666329e1a4469c93d889c102a15dc9 => generated 0 bytes in 509 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:01.894113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c41d2fe78f3c4f3cbb71565656e3548c', 'request_id': 'req-b3282fad-e47e-49ad-88c2-19991500aa18'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.906594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c41d2fe78f3c4f3cbb71565656e3548c', 'request_id': 'req-b3282fad-e47e-49ad-88c2-19991500aa18'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.924403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] The token cache is being invalidate because project c41d2fe78f3c4f3cbb71565656e3548c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:01.924704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.925380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b3282fad-e47e-49ad-88c2-19991500aa18 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:01.930843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 931/1849] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:01 2022] DELETE /identity/v3/projects/c41d2fe78f3c4f3cbb71565656e3548c => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:01.933168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:01.933462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:01.933733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:01.968447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:01.970008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:02.106008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 919/1850] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:02.113817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.116812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.116812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.139308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:02.141047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:02.149941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 932/1851] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:02.156986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.166825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f0a266-ae1f-4a14-b369-043b953f12d0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.171513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.172223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.172504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.172780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.173431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:02.173739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.175459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.188682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 920/1852] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:02.193171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.203032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3515a91a-2e08-4edf-bd15-726506dad1e2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.208518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.217010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.217010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.217010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.217010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.217010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.249088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.250858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:02.274974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-569546e3-f246-403c-bb7b-e2ceb6dabed1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.293115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.295009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.295445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.295727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] PATH_INFO: `/v3/projects/253f3ef5ee774e92b68232ac728f7f76` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.296946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=253f3ef5ee774e92b68232ac728f7f76)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.299292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.307621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 933/1853] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:02 2022] POST /identity/v3/projects => generated 365 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:02.318791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.393945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e7dcba0-3425-4dc2-9908-36016595e2e9 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.398815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.399461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.399745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.400023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.401288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.403347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.461238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '253f3ef5ee774e92b68232ac728f7f76', 'request_id': 'req-1be6e4a7-bd44-4016-a559-223bee9ebfa8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.467375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '253f3ef5ee774e92b68232ac728f7f76', 'request_id': 'req-1be6e4a7-bd44-4016-a559-223bee9ebfa8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.559484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 934/1854] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:02 2022] POST /identity/v3/users => generated 283 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:02.564345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.566729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:02.574071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '253f3ef5ee774e92b68232ac728f7f76', 'request_id': 'req-1be6e4a7-bd44-4016-a559-223bee9ebfa8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.584037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3af5ffe1-cfb7-42ff-ac2f-8ea34d245382 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.596025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] The token cache is being invalidate because project 253f3ef5ee774e92b68232ac728f7f76 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:02.597622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.598891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.601809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 921/1855] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:02 2022] DELETE /identity/v3/projects/253f3ef5ee774e92b68232ac728f7f76 => generated 0 bytes in 353 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:02.603354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.606351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:02.685858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.686729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.686958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.687105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] PATH_INFO: `/v3/projects/8b28c20d84ed481f9f7a3a088ced3bbf` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.688545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8b28c20d84ed481f9f7a3a088ced3bbf)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.690877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.713808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1be6e4a7-bd44-4016-a559-223bee9ebfa8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.720152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.720906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.721305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.721649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.722361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.725253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.750987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 922/1856] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:02 2022] GET /identity/v3/roles => generated 1420 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:02.752492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:02.754696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:02.772162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4487255-3757-4667-90a7-6600f0de4941 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:02.777208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:02.777846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:02.778124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:02.778397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] PATH_INFO: `/v3/users/a48af238f391411e8d08acce64fa92e6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:02.779757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a48af238f391411e8d08acce64fa92e6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:02.781915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:02.889504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8b28c20d84ed481f9f7a3a088ced3bbf', 'request_id': 'req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.985496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8b28c20d84ed481f9f7a3a088ced3bbf', 'request_id': 'req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:02.999940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8b28c20d84ed481f9f7a3a088ced3bbf', 'request_id': 'req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.028616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] The token cache is being invalidate because project 8b28c20d84ed481f9f7a3a088ced3bbf was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:03.029040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.029888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.036028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 935/1857] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:02 2022] DELETE /identity/v3/projects/8b28c20d84ed481f9f7a3a088ced3bbf => generated 0 bytes in 468 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:03.037299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:03.038821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.093250 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a48af238f391411e8d08acce64fa92e6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.100553 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a48af238f391411e8d08acce64fa92e6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.101297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a48af238f391411e8d08acce64fa92e6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.101970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a48af238f391411e8d08acce64fa92e6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.110486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45ec79b4-c3cf-41f8-8426-39f782c3a5d6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:03.116063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:03.116721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.117050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.117324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.117971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:03.120695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:03.131116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 936/1858] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:03 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:03.131958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:03.133641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.146639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a94557-cf46-4625-a734-66185d1de8f2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:03.151735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:03.152364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.152641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.152911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] PATH_INFO: `/v3/projects/a67c9515ecca4ed68247ed389a5870c6` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.154069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a67c9515ecca4ed68247ed389a5870c6)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:03.156016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:03.179911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a48af238f391411e8d08acce64fa92e6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.201457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 923/1859] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:02 2022] DELETE /identity/v3/users/a48af238f391411e8d08acce64fa92e6 => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:03.202630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:03.204370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.278691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa99b3f5-c990-4dbc-9fd4-5f2017196287 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:03.283678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:03.284319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.284763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.285048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] PATH_INFO: `/v3/users/9834860f5d464a5f99fc5ad8dcca2c06` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.286191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9834860f5d464a5f99fc5ad8dcca2c06)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:03.288405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:03.317187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a67c9515ecca4ed68247ed389a5870c6', 'request_id': 'req-39487305-84dc-4d8d-8078-2f40afe8b19f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.436959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a67c9515ecca4ed68247ed389a5870c6', 'request_id': 'req-39487305-84dc-4d8d-8078-2f40afe8b19f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.459498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a67c9515ecca4ed68247ed389a5870c6', 'request_id': 'req-39487305-84dc-4d8d-8078-2f40afe8b19f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.471753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] The token cache is being invalidate because project a67c9515ecca4ed68247ed389a5870c6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:03.472035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.472711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.474245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 937/1860] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:03 2022] DELETE /identity/v3/projects/a67c9515ecca4ed68247ed389a5870c6 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:03.497345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:03.503969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.517720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9834860f5d464a5f99fc5ad8dcca2c06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.528125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9834860f5d464a5f99fc5ad8dcca2c06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.529088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9834860f5d464a5f99fc5ad8dcca2c06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.529900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9834860f5d464a5f99fc5ad8dcca2c06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.586368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39487305-84dc-4d8d-8078-2f40afe8b19f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:03.591225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95c6900-c81f-475a-9d94-0e1ae31796b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:03.591886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95c6900-c81f-475a-9d94-0e1ae31796b7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.592164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95c6900-c81f-475a-9d94-0e1ae31796b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.592550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95c6900-c81f-475a-9d94-0e1ae31796b7 admin admin] PATH_INFO: `/v3/projects/78b55416bb584f8aa9992d308e1b1e1b/users/60766b0ad9f549ef8eb76064065fffd7/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.608120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95c6900-c81f-475a-9d94-0e1ae31796b7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=78b55416bb584f8aa9992d308e1b1e1b, user_id=60766b0ad9f549ef8eb76064065fffd7, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:03.610841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95c6900-c81f-475a-9d94-0e1ae31796b7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:03.636103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-08357f19-0f53-4bf7-afc6-d538a9c5b0e3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9834860f5d464a5f99fc5ad8dcca2c06'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:03.655464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 924/1861] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:03 2022] DELETE /identity/v3/users/9834860f5d464a5f99fc5ad8dcca2c06 => generated 0 bytes in 453 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:03.657101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac05e484-dfc5-46ee-9f5c-84b52c6d5879 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.657651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac05e484-dfc5-46ee-9f5c-84b52c6d5879 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.658095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac05e484-dfc5-46ee-9f5c-84b52c6d5879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.683402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ac05e484-dfc5-46ee-9f5c-84b52c6d5879 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:03.685301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ac05e484-dfc5-46ee-9f5c-84b52c6d5879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.723181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 938/1862] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:03 2022] PUT /identity/v3/projects/78b55416bb584f8aa9992d308e1b1e1b/users/60766b0ad9f549ef8eb76064065fffd7/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:03.726737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf257243-7227-4f66-8434-d68d64f0fdd2 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.726737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf257243-7227-4f66-8434-d68d64f0fdd2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.726737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf257243-7227-4f66-8434-d68d64f0fdd2 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.726737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 939/1863] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:37:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:37:03.736423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf257243-7227-4f66-8434-d68d64f0fdd2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:03.738328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bf257243-7227-4f66-8434-d68d64f0fdd2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.794086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 925/1864] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:03.794086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d431-e207-49c5-bb16-0d213207b3ea None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.794086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d431-e207-49c5-bb16-0d213207b3ea None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.794086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f8d431-e207-49c5-bb16-0d213207b3ea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.847545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf257243-7227-4f66-8434-d68d64f0fdd2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:03.851665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-55f8d431-e207-49c5-bb16-0d213207b3ea None None] MFA Rules not processed for user `60766b0ad9f549ef8eb76064065fffd7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:03.853150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-55f8d431-e207-49c5-bb16-0d213207b3ea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.855483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:03.856518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.857015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.857527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] PATH_INFO: `/v3/projects/a50029e7d2984db5940577455e88f534` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.860719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] RBAC: Authorizing `identity:delete_project(project_id=a50029e7d2984db5940577455e88f534)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:03.864467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:03.945480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:03.947819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 926/1865] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:03 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:03.949402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68590e6e-89ff-4462-9bc3-dc2269d7c775 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.949691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68590e6e-89ff-4462-9bc3-dc2269d7c775 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.950116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68590e6e-89ff-4462-9bc3-dc2269d7c775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.971649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-68590e6e-89ff-4462-9bc3-dc2269d7c775 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:03.972893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-68590e6e-89ff-4462-9bc3-dc2269d7c775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:03.986095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 927/1866] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:03.986095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:03.986095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:03.986095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:03.995068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a50029e7d2984db5940577455e88f534', 'request_id': 'req-db5043c3-2a5c-4d42-9283-f9a9010fca4f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.008015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:04.009267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.074901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a50029e7d2984db5940577455e88f534', 'request_id': 'req-db5043c3-2a5c-4d42-9283-f9a9010fca4f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.085294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a50029e7d2984db5940577455e88f534', 'request_id': 'req-db5043c3-2a5c-4d42-9283-f9a9010fca4f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.094448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 928/1867] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:37:03 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:04.095361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.096933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.108862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] The token cache is being invalidate because project a50029e7d2984db5940577455e88f534 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:04.110919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.114744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.114744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 940/1868] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:03 2022] DELETE /identity/v3/projects/a50029e7d2984db5940577455e88f534 => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:04.114744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.124585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.190675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b924942-1945-4bfe-825d-828e3e88025f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.196270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.196886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.197166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.197488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.198913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.200478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.203576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5043c3-2a5c-4d42-9283-f9a9010fca4f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.208281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.208877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.209156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.209424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.211097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:04.211405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.213672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1846aba-5447-4ea8-85b9-9a27ac852b9a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.226571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 941/1869] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 112 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:04.227948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.228231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.228505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.249700 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:04.250961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.259292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 942/1870] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:04.260117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.261374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.275158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b1f2e4c-495e-4de7-8c9d-1d348120c6b3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.280511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.281135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.281476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.283403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] PATH_INFO: `/v3/users/5c3cefb475384e4e9936badbfb10295d` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.283403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] RBAC: Authorizing `identity:delete_user(user_id=5c3cefb475384e4e9936badbfb10295d)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.285187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.293902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 929/1871] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/projects => generated 365 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:04.294732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.375684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6306f995-7ca8-4761-a326-efe84a543f21 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.384878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31d8397f-9348-45be-add0-b73c6343192c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.386526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8397f-9348-45be-add0-b73c6343192c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.386951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8397f-9348-45be-add0-b73c6343192c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.387395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31d8397f-9348-45be-add0-b73c6343192c admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.388226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d8397f-9348-45be-add0-b73c6343192c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.395096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31d8397f-9348-45be-add0-b73c6343192c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.506483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5c3cefb475384e4e9936badbfb10295d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.507300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5c3cefb475384e4e9936badbfb10295d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.514945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5c3cefb475384e4e9936badbfb10295d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.515794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5c3cefb475384e4e9936badbfb10295d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.524016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 930/1872] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/projects => generated 387 bytes in 229 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:04.525320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.526270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.526490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.546549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:04.547790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.602422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5c3cefb475384e4e9936badbfb10295d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:04.636621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 943/1873] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:04 2022] DELETE /identity/v3/users/5c3cefb475384e4e9936badbfb10295d => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:04.636621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.639835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.655784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 931/1874] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:04.660582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.660582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.731723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02223663-bc6f-4ae2-afbe-c501db8a33eb admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.736773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.737387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.737658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.737923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.738554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.740567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.754026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82671b08-d99b-4e96-8fbf-37e0cf994b23 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.817881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.818518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.818793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.819078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.820819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.831216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 944/1875] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/users => generated 289 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:04.832112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.833747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.836958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.841290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.847310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebaa9dc0-78de-485c-b32b-d22c5a9fbc93 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.851930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d2217b-ba11-4000-a789-77dfea4e0262 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.852528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d2217b-ba11-4000-a789-77dfea4e0262 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.852800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d2217b-ba11-4000-a789-77dfea4e0262 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.853065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d2217b-ba11-4000-a789-77dfea4e0262 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.853904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d2217b-ba11-4000-a789-77dfea4e0262 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.855848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d2217b-ba11-4000-a789-77dfea4e0262 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.860260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 932/1876] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:04 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:04.861452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.862932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.877094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5888ecd3-4d1c-4c37-99ff-7a0028ee7803 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:04.882431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:04.883179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.883475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.883732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] PATH_INFO: `/v3/projects/1222eb00196740459dd73461e1dcb9b6` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.885402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1222eb00196740459dd73461e1dcb9b6)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:04.887562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:04.958961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 945/1877] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/users => generated 294 bytes in 124 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:04.958961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:04.958961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:04.958961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:04.976754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:04.979279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:04.986989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 946/1878] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:04 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:04.992900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:04.999265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70c76ccd-d55d-4feb-835f-104fbce6ce0b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.001334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1222eb00196740459dd73461e1dcb9b6', 'request_id': 'req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:05.007633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1222eb00196740459dd73461e1dcb9b6', 'request_id': 'req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:05.060262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:05.069642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.071312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.071312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.071312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.071637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:05.071957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.074091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.086005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 947/1879] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:05.086900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.091265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1222eb00196740459dd73461e1dcb9b6', 'request_id': 'req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:05.097090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bba40f9-a069-4f43-ac0c-af9ae4509de6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.102144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.102688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.102999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.103288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.104780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.105860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.106012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] The token cache is being invalidate because project 1222eb00196740459dd73461e1dcb9b6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:05.106433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:05.107211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:05.108813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 933/1880] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:04 2022] DELETE /identity/v3/projects/1222eb00196740459dd73461e1dcb9b6 => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:05.109761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.111155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.121145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 948/1881] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:05.121145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.140838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.141142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12103d-48bf-4ed2-8cfe-6d6e8fa36f48 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.141142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.141142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.141142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.141467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.141467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.141467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.148613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 934/1882] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:05.149648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.151608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.166343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52040c54-9284-4093-ac10-2013891d7cfa admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.176414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.176414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.176414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.176414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.176947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.176947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.191559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce47793-a4ca-49a2-b24a-ab8c04e17c7c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.195345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.195959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.196229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.196630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.199762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.212957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.214622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d76f5c38-0d60-4097-bd11-9ba3876797d2 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.226118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 949/1883] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:05.227412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.227774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.228021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.248998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:05.250406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.261647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 950/1884] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:05.262763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.264469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.275601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 935/1885] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:05 2022] POST /identity/v3/projects => generated 375 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:05.278678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.282604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df6a0916-2490-4fa8-a61c-75f897c8ae0f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.386929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.387620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.387900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.388171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] PATH_INFO: `/v3/projects/9545e05876d14ef08ab58ebd18bf3103/users/2554247492024c7da62c0aae89b7a70b/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.410656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36d83b64-528a-4951-8022-fd1b3f330513 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.411831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] RBAC: Authorizing `identity:create_grant(project_id=9545e05876d14ef08ab58ebd18bf3103, user_id=2554247492024c7da62c0aae89b7a70b, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.418014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-887c8d87-b6f4-448c-abbf-7fa91b3f5c70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.419347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c8d87-b6f4-448c-abbf-7fa91b3f5c70 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.419642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c8d87-b6f4-448c-abbf-7fa91b3f5c70 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.419916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-887c8d87-b6f4-448c-abbf-7fa91b3f5c70 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.420596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c8d87-b6f4-448c-abbf-7fa91b3f5c70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.422216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-887c8d87-b6f4-448c-abbf-7fa91b3f5c70 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.427252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.437540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 936/1886] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:05.437540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.437540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.437540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.463635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:05.466018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:05.478408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 937/1887] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 41 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:05.478408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.488664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182c753d-3d86-48bc-a9ed-a89882e0b4f4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.493817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.494473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.495115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.495115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.495645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.497731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.540854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 951/1888] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:05 2022] PUT /identity/v3/projects/9545e05876d14ef08ab58ebd18bf3103/users/2554247492024c7da62c0aae89b7a70b/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:05.542144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.595733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 938/1889] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:05 2022] POST /identity/v3/users => generated 288 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:05.595733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.618739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1994a7a8-74cf-4120-a9af-f4c8bd8c7f4b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.649312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-261cd073-84f0-4ed7-8c50-82d26179beba admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.655074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.655768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.656045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.656352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] PATH_INFO: `/v3/projects/e46b3584f71c4e6a8f8c4880ca6b6a6a/users/da408e216c424a9e9ca4bab3a792e727/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.674467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e46b3584f71c4e6a8f8c4880ca6b6a6a, user_id=da408e216c424a9e9ca4bab3a792e727, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.677798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.704124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.704773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.705048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.705640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.705970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:05.706272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.707944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.726552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 939/1890] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:05.726552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.752311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0ec4e51-0f74-4d97-94d3-65ddd33543b5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.763921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.766669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.766669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.766669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.767991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.770034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.788492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 952/1891] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:05 2022] PUT /identity/v3/projects/e46b3584f71c4e6a8f8c4880ca6b6a6a/users/da408e216c424a9e9ca4bab3a792e727/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:05.789554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.803502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 940/1892] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 79 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:05.804401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:05.824753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ed7ad23-23ca-48be-a081-a13b5b136735 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.882338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d4e3460-2491-4ea8-93ee-41c8c94adea4 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:05.891466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30f84441-9c5e-485e-972c-7ace0610bf73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.892236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f84441-9c5e-485e-972c-7ace0610bf73 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.893066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f84441-9c5e-485e-972c-7ace0610bf73 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.893343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30f84441-9c5e-485e-972c-7ace0610bf73 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.894838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f84441-9c5e-485e-972c-7ace0610bf73 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.896605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30f84441-9c5e-485e-972c-7ace0610bf73 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.911887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 953/1893] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:05 2022] GET /identity/v3/roles => generated 1420 bytes in 120 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:05.911887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.911887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.911887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.955351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:05.956091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:05.956438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:05.956784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:05.957680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:05.959541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] MFA Rules not processed for user `da408e216c424a9e9ca4bab3a792e727`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:05.959716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:05.960830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.029684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 954/1894] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:05 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:06.030519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.041556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d94c7a1-18ec-4e5f-b4a5-aa0377a70a91 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.046904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.047603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.047883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.048155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] PATH_INFO: `/v3/projects/9545e05876d14ef08ab58ebd18bf3103/users/2554247492024c7da62c0aae89b7a70b/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.050105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9545e05876d14ef08ab58ebd18bf3103, user_id=2554247492024c7da62c0aae89b7a70b, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.052772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.064794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 941/1895] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:05 2022] POST /identity/v3/projects => generated 367 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:06.064794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.065043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 955/1896] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:06 2022] PUT /identity/v3/projects/9545e05876d14ef08ab58ebd18bf3103/users/2554247492024c7da62c0aae89b7a70b/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:06.065141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.065418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.130465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-318b0a4a-2eaa-48db-8c35-634ad0145766 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.135263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.135896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.136177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.136453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] PATH_INFO: `/v3/projects/cc44eab05e2a4639a54f43a3212dee90` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.136844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b92aa0f-9b00-46e9-a833-c8b10468bab9 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.137564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc44eab05e2a4639a54f43a3212dee90)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.139536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.141479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.142077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.142356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.142626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.143964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.145675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.156178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 956/1897] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:06.157021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.168411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c90e179-8bfc-4059-acfa-c02d1447996c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.173256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.173850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.174133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.174400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.175071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.176974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.279286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 957/1898] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:06 2022] POST /identity/v3/projects => generated 387 bytes in 123 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:06.280155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.289609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc44eab05e2a4639a54f43a3212dee90', 'request_id': 'req-6aac1985-b7d6-4c04-a837-30e2fd8b1693'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.295910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc44eab05e2a4639a54f43a3212dee90', 'request_id': 'req-6aac1985-b7d6-4c04-a837-30e2fd8b1693'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.343212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae0027e1-a71c-407c-8f7b-1fce0bd1d378 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.348106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.348716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.348999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.349345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.350107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.352075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.379650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc44eab05e2a4639a54f43a3212dee90', 'request_id': 'req-6aac1985-b7d6-4c04-a837-30e2fd8b1693'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.391945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] The token cache is being invalidate because project cc44eab05e2a4639a54f43a3212dee90 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:06.392174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.392855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6aac1985-b7d6-4c04-a837-30e2fd8b1693 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.394374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 942/1899] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:06 2022] DELETE /identity/v3/projects/cc44eab05e2a4639a54f43a3212dee90 => generated 0 bytes in 332 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:06.395769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.396046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.396583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.424200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] MFA Rules not processed for user `2554247492024c7da62c0aae89b7a70b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:06.425397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.442797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 958/1900] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:06 2022] POST /identity/v3/users => generated 284 bytes in 163 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:06.443436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.444982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.457602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b67879-b960-4dc7-a5a1-a0f64af1f312 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.462446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.463653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.463955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.464234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] PATH_INFO: `/v3/projects/6abe722eb0a34e98ac59acac0b02c903/users/653e9deedcac44438d7a7535e7c433d6/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.479673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6abe722eb0a34e98ac59acac0b02c903, user_id=653e9deedcac44438d7a7535e7c433d6, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.482424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.498033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 943/1901] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:06 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:06.498930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.500229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.512786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71107ed9-1e6f-43ac-a5ac-e32df7f384cf None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.517280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.517866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.518140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.518407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.519089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.520997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.567901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 959/1902] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:06 2022] PUT /identity/v3/projects/6abe722eb0a34e98ac59acac0b02c903/users/653e9deedcac44438d7a7535e7c433d6/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 125 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:06.568874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.570422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.612137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 944/1903] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:06 2022] POST /identity/v3/users => generated 300 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:06.612137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.612137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.624317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0cda954-3488-4081-8dd4-787c987cbb24 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.635390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef482ca-879a-43bb-b630-72047bc6fde2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.639994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.640643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.640973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.641196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] PATH_INFO: `/v3/users/b89a4993963f48e19a3fea58cf864017` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.647969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b89a4993963f48e19a3fea58cf864017)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.647969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.682417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.683055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.683334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.683605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.684268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.685860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.695441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 945/1904] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:06.696363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.698096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.752819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1be00dd-fc24-42c2-a441-2b93b2b03753 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.756507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60db013-af77-4f5d-80ee-4266f120fae2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.757149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60db013-af77-4f5d-80ee-4266f120fae2 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.757424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60db013-af77-4f5d-80ee-4266f120fae2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.757696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60db013-af77-4f5d-80ee-4266f120fae2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.772701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:06.831421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60db013-af77-4f5d-80ee-4266f120fae2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.831421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60db013-af77-4f5d-80ee-4266f120fae2 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:06.840625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 946/1905] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:06 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:06.841941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.842230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.842504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.877068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] MFA Rules not processed for user `653e9deedcac44438d7a7535e7c433d6`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:06.879299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.920767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b89a4993963f48e19a3fea58cf864017'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.921534 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b89a4993963f48e19a3fea58cf864017'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.931566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b89a4993963f48e19a3fea58cf864017'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.932411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b89a4993963f48e19a3fea58cf864017'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:06.962455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 947/1906] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:06 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:06.964017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:06.965436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:06.981283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8315ea0-eb09-48b3-93a3-959ff0e3a4e3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:06.987282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:06.987975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:06.988327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:06.988676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:06.989387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:06.991121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.004898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 948/1907] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:06 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:07.004898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.007034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.043752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6933ddb2-c169-4ff7-86dd-cc667db63448 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.050287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b89a4993963f48e19a3fea58cf864017'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:07.056041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.058802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.058802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.058802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.058802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.059894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.086388 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 949/1908] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:07.088063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.092281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.096733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 960/1909] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:06 2022] DELETE /identity/v3/users/b89a4993963f48e19a3fea58cf864017 => generated 0 bytes in 528 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:07.099892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.185792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9428626-825a-460e-9506-151149fbe597 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.197662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.199146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.199479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.199853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] PATH_INFO: `/v3/projects/b7f9d8b6e0c742e1bf847f2eca04c196/users/554206fce1ab40589f6af0fc35f92207/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.218873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7f9d8b6e0c742e1bf847f2eca04c196, user_id=554206fce1ab40589f6af0fc35f92207, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.221870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.262093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763fa072-b511-4faa-8c34-75567cef6989 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.314696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.315464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.315623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.315901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.361286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 961/1910] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:07 2022] PUT /identity/v3/projects/b7f9d8b6e0c742e1bf847f2eca04c196/users/554206fce1ab40589f6af0fc35f92207/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 264 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:07.362266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.396509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.398115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.409918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 950/1911] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:07 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:07.410817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.412636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.445247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dee40de3-98d0-4035-9b1a-2803f62d7ee3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.457888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.458643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.459004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.463929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] PATH_INFO: `/v3/projects/9da58a8167844d55826b167bfd25f960/users/b4fcf1c40e0c4a05a6622ef6d92fb452/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.489640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9da58a8167844d55826b167bfd25f960, user_id=b4fcf1c40e0c4a05a6622ef6d92fb452, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.493223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.504308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3f481c3-a936-4e83-af83-5a599d1e7a85 service glance] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.581660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 962/1912] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:07 2022] PUT /identity/v3/projects/9da58a8167844d55826b167bfd25f960/users/b4fcf1c40e0c4a05a6622ef6d92fb452/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:07.581660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.582895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.583539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.583821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.584096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.585998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.674045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d299c818-4820-49d9-996a-3e9eb2ea76e1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.687427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1712da5d-325e-45a5-b684-14bdb83606e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.688401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1712da5d-325e-45a5-b684-14bdb83606e3 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.688718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1712da5d-325e-45a5-b684-14bdb83606e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.689009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1712da5d-325e-45a5-b684-14bdb83606e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.689758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1712da5d-325e-45a5-b684-14bdb83606e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.691545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1712da5d-325e-45a5-b684-14bdb83606e3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.696610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.698155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.702864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 963/1913] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:07 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:07.704390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.704678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.704943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.712021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 951/1914] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:07 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 301 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:07.713014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.725684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-252a1c28-00f3-47ca-9a57-2da76acffc5d service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.733227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbf893b1-0267-482d-8868-7a284f0b5161 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.734047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf893b1-0267-482d-8868-7a284f0b5161 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.734403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf893b1-0267-482d-8868-7a284f0b5161 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.734802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbf893b1-0267-482d-8868-7a284f0b5161 admin admin] PATH_INFO: `/v3/projects/b7f9d8b6e0c742e1bf847f2eca04c196/users/554206fce1ab40589f6af0fc35f92207/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.737314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf893b1-0267-482d-8868-7a284f0b5161 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b7f9d8b6e0c742e1bf847f2eca04c196, user_id=554206fce1ab40589f6af0fc35f92207, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.741539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbf893b1-0267-482d-8868-7a284f0b5161 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.756210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 952/1915] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:07 2022] PUT /identity/v3/projects/b7f9d8b6e0c742e1bf847f2eca04c196/users/554206fce1ab40589f6af0fc35f92207/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 43 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:07.764913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] MFA Rules not processed for user `b4fcf1c40e0c4a05a6622ef6d92fb452`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:07.764913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.767288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.767410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.768036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.814910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] MFA Rules not processed for user `554206fce1ab40589f6af0fc35f92207`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:07.816302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.865754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 964/1916] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:07 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:07.866687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.868009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:07.894549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 953/1917] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:07 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:07.895592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:07.931039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:07.939286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b05069f0-71d5-478a-a6ce-3a0107be278c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.944408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.945037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.945318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.945593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] PATH_INFO: `/v3/projects/762b18ce44b845238df5e7a6036e5743` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.946706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] RBAC: Authorizing `identity:delete_project(project_id=762b18ce44b845238df5e7a6036e5743)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.949453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:07.969197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae1f5eed-93e6-41fb-b0db-4a4f6a093e54 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:07.974162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:07.974796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:07.975071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:07.975342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:07.975972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:07.977922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.098259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '762b18ce44b845238df5e7a6036e5743', 'request_id': 'req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.122573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 954/1918] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:07 2022] POST /identity/v3/projects => generated 367 bytes in 227 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:08.126539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.147061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '762b18ce44b845238df5e7a6036e5743', 'request_id': 'req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.156351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '762b18ce44b845238df5e7a6036e5743', 'request_id': 'req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.171265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] The token cache is being invalidate because project 762b18ce44b845238df5e7a6036e5743 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:08.171665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.172679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.177660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 965/1919] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:07 2022] DELETE /identity/v3/projects/762b18ce44b845238df5e7a6036e5743 => generated 0 bytes in 308 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:08.178878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.180375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.212683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0644e14a-e57c-455f-b364-30d52d1874a3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.219644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.220252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.220632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.221397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.222034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.224773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.265284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfcf2c2-78bd-4eb5-b3e8-18be1d0bec77 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.284324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.285255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.285536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.285806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] PATH_INFO: `/v3/users/e393b225194347f090862ba815750a90` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.287253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e393b225194347f090862ba815750a90)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.290145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.388156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 955/1920] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:08 2022] POST /identity/v3/users => generated 290 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:08.389036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.391205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.448747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a98b5b9c-1133-47f6-9309-be3407534869 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.452797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.453407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.453682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.453947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.455700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.517152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e393b225194347f090862ba815750a90'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.517821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e393b225194347f090862ba815750a90'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.526197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e393b225194347f090862ba815750a90'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.526899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e393b225194347f090862ba815750a90'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.543562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.545002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.555035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 956/1921] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:08 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:08.555919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.557342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.572230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cbc682e-c9c7-4db7-8b08-7b6f99d9e3e8 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.578485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.578485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.578485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.578737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.579399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.581602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.597271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 957/1922] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:08 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:08.597271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.597271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.610668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5836bc-35a3-4319-84ea-33b64beeece3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.615561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.616195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.616481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.616759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] PATH_INFO: `/v3/users/da408e216c424a9e9ca4bab3a792e727` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.617975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] RBAC: Authorizing `identity:delete_user(user_id=da408e216c424a9e9ca4bab3a792e727)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.619973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.654692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e393b225194347f090862ba815750a90'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.688247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 966/1923] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:08 2022] DELETE /identity/v3/users/e393b225194347f090862ba815750a90 => generated 0 bytes in 510 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:08.689245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.690788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.740909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:08.761273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14152f2a-9d4f-4f0a-85c8-3bd5b87ecc33 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.766655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.767435 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.767712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.768358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] PATH_INFO: `/v3/projects/0a2a6cfbd0954ecb8aac28c00f3d39e4/users/297ed9c9f63a42fda82b0fd4a25e6eae/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.785210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a2a6cfbd0954ecb8aac28c00f3d39e4, user_id=297ed9c9f63a42fda82b0fd4a25e6eae, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.788208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.839478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'da408e216c424a9e9ca4bab3a792e727'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.846500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'da408e216c424a9e9ca4bab3a792e727'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.847203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'da408e216c424a9e9ca4bab3a792e727'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.847822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'da408e216c424a9e9ca4bab3a792e727'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.866650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 967/1924] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:08 2022] PUT /identity/v3/projects/0a2a6cfbd0954ecb8aac28c00f3d39e4/users/297ed9c9f63a42fda82b0fd4a25e6eae/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:08.866650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.869335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:08.944095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3b4e436-31ab-4f9b-ab61-e1b708a3946d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.948719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.949405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.950222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.950492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.951174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.952778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:08.969633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 968/1925] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:08 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:08.969633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:08.970019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-073a8385-f3b2-4093-af4a-1417fefc3e35 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'da408e216c424a9e9ca4bab3a792e727'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:08.977908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3b525c1-0e19-4014-bf6a-796db88df2c2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:08.984767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab56cb89-b118-4d21-9355-e0a23403b70f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:08.985660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56cb89-b118-4d21-9355-e0a23403b70f admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:08.986024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56cb89-b118-4d21-9355-e0a23403b70f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:08.986730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab56cb89-b118-4d21-9355-e0a23403b70f admin admin] PATH_INFO: `/v3/projects/0a2a6cfbd0954ecb8aac28c00f3d39e4/users/297ed9c9f63a42fda82b0fd4a25e6eae/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:08.988735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab56cb89-b118-4d21-9355-e0a23403b70f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a2a6cfbd0954ecb8aac28c00f3d39e4, user_id=297ed9c9f63a42fda82b0fd4a25e6eae, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:08.992160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab56cb89-b118-4d21-9355-e0a23403b70f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:09.004314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 969/1927] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:08 2022] PUT /identity/v3/projects/0a2a6cfbd0954ecb8aac28c00f3d39e4/users/297ed9c9f63a42fda82b0fd4a25e6eae/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:09.004568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 958/1927] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:08 2022] DELETE /identity/v3/users/da408e216c424a9e9ca4bab3a792e727 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:09.008344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.008750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.009120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.016308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.016308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.016565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.030559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:09.031956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.049555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] MFA Rules not processed for user `297ed9c9f63a42fda82b0fd4a25e6eae`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:09.050861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.097637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 970/1928] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:09.112504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:09.114237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.140782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 959/1929] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:09 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:09.162927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:09.166801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.184751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde5b68d-55e6-4aae-8771-7ef811a5168b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:09.189004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:09.194936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.194936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.194936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.206741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:09.208063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fffc6885-584c-4e55-9ab1-2db1cb030b39 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:09.208797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:09.217620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:09.218491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.220944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.220944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] PATH_INFO: `/v3/projects/96955b3b25024a7493f2d5cd1357c814` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.220944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] RBAC: Authorizing `identity:delete_project(project_id=96955b3b25024a7493f2d5cd1357c814)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:09.222939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 971/1930] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:09 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:09.223056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:09.224064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:09.225753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.322213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fede88fa-ad8b-4a66-aba9-244955e70b48 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:09.328834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d005c9e-0806-4368-ad1b-a77ad151ddf4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:09.329448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d005c9e-0806-4368-ad1b-a77ad151ddf4 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.329844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d005c9e-0806-4368-ad1b-a77ad151ddf4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.331009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d005c9e-0806-4368-ad1b-a77ad151ddf4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.368304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d005c9e-0806-4368-ad1b-a77ad151ddf4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:09.374593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d005c9e-0806-4368-ad1b-a77ad151ddf4 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:09.376658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96955b3b25024a7493f2d5cd1357c814', 'request_id': 'req-fb900074-27fd-4fa7-a8ea-f8800606e21f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:09.387367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 972/1931] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:09 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:09.398799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96955b3b25024a7493f2d5cd1357c814', 'request_id': 'req-fb900074-27fd-4fa7-a8ea-f8800606e21f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:09.537883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96955b3b25024a7493f2d5cd1357c814', 'request_id': 'req-fb900074-27fd-4fa7-a8ea-f8800606e21f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:09.549973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] The token cache is being invalidate because project 96955b3b25024a7493f2d5cd1357c814 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:09.551790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:09.553201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:09.559209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 960/1932] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:09 2022] DELETE /identity/v3/projects/96955b3b25024a7493f2d5cd1357c814 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:09.560319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a56e84c-0289-4fb5-848b-d6f32ffe2846 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.560607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a56e84c-0289-4fb5-848b-d6f32ffe2846 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.560883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a56e84c-0289-4fb5-848b-d6f32ffe2846 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.591317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6a56e84c-0289-4fb5-848b-d6f32ffe2846 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:09.592305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6a56e84c-0289-4fb5-848b-d6f32ffe2846 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.761544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:09.763725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.781439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 973/1933] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 222 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:09.789014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.789431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.790511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.850960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:09.852966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.864273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 974/1934] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:09 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:09.874396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:09.886974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c12e27c-fd21-4d48-9f62-878532e0d84b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:09.892258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:09.893439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.893439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.893439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.894401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:09.894570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:09.896618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:09.909518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb900074-27fd-4fa7-a8ea-f8800606e21f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:09.914628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 975/1935] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:09.914820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:09.915473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.915667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:09.915827 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.916234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.919664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:09.926944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d889582d-5043-49c7-a213-1b02df83c7c2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:09.931928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:09.932544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:09.932823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:09.933091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:09.934369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:09.936343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.001299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.002948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.013698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 961/1936] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:09 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:10.014406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.015782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:10.034847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c51863cd-5803-4442-b3d6-f9139010665c service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.046559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.047595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.047947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.048288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] PATH_INFO: `/v3/projects/e46b3584f71c4e6a8f8c4880ca6b6a6a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.049746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e46b3584f71c4e6a8f8c4880ca6b6a6a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.051820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.082599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 976/1937] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:09 2022] POST /identity/v3/projects => generated 369 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:10.083518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.155570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb4ad272-d77c-4ca4-96fb-f82a351ac2de admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.160934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.169374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.169374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.169374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.169374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.169374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.200851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e46b3584f71c4e6a8f8c4880ca6b6a6a', 'request_id': 'req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.206250 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e46b3584f71c4e6a8f8c4880ca6b6a6a', 'request_id': 'req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.287235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e46b3584f71c4e6a8f8c4880ca6b6a6a', 'request_id': 'req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.296379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 977/1938] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:10 2022] POST /identity/v3/users => generated 285 bytes in 212 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:10.298985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.301808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] The token cache is being invalidate because project e46b3584f71c4e6a8f8c4880ca6b6a6a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:10.302300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.303134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.304845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 962/1939] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:10 2022] DELETE /identity/v3/projects/e46b3584f71c4e6a8f8c4880ca6b6a6a => generated 0 bytes in 291 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:10.310346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.311796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:10.322287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1d35cb0-b54e-475b-a752-76edb00d5e06 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.388312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.389032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.389502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.389877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.390074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d9f3ac-f909-4f52-9fc9-8cd09cbdbcf3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.390811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.392699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.394810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.395533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.395759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.396033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] PATH_INFO: `/v3/users/554206fce1ab40589f6af0fc35f92207` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.397149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] RBAC: Authorizing `identity:delete_user(user_id=554206fce1ab40589f6af0fc35f92207)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.399127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.406201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 978/1940] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:10 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:10.411956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.412608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:10.426940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9407a573-eca2-4362-b1db-eefeb8da34e5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.432632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.433237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.433515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.434367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.435067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.436730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.446966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 979/1941] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:10 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:10.451950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.462203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f4034e7-1c2c-47fc-a801-52718c147da0 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.467259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a736bd0b-a6ac-4464-970f-aec8b0f1be9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.467910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a736bd0b-a6ac-4464-970f-aec8b0f1be9a admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.468220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a736bd0b-a6ac-4464-970f-aec8b0f1be9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.468491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a736bd0b-a6ac-4464-970f-aec8b0f1be9a admin admin] PATH_INFO: `/v3/projects/69108d4306174c54ad19214c61b21748/users/7a840f777cf24fda888ae3cfc5293941/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.482933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a736bd0b-a6ac-4464-970f-aec8b0f1be9a admin admin] RBAC: Authorizing `identity:create_grant(project_id=69108d4306174c54ad19214c61b21748, user_id=7a840f777cf24fda888ae3cfc5293941, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:10.485614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a736bd0b-a6ac-4464-970f-aec8b0f1be9a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:10.518167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:10.585410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 980/1942] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:10 2022] PUT /identity/v3/projects/69108d4306174c54ad19214c61b21748/users/7a840f777cf24fda888ae3cfc5293941/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:10.596231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.596851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.596851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:10.616687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:10.637814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '554206fce1ab40589f6af0fc35f92207'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.662847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '554206fce1ab40589f6af0fc35f92207'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.665453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] MFA Rules not processed for user `7a840f777cf24fda888ae3cfc5293941`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:10.665805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '554206fce1ab40589f6af0fc35f92207'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.666633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '554206fce1ab40589f6af0fc35f92207'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.666766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:10.774274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '554206fce1ab40589f6af0fc35f92207'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:10.804846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 981/1943] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:10 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:10.815394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 963/1944] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:10 2022] DELETE /identity/v3/users/554206fce1ab40589f6af0fc35f92207 => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:10.835695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:10.835695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:10.928908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae5cec55-0426-4d16-9581-90cd5975244f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:10.998376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6ff29bf-1d39-4e64-8319-67465096aff0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:10.999158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ff29bf-1d39-4e64-8319-67465096aff0 service glance] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:10.999499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ff29bf-1d39-4e64-8319-67465096aff0 service glance] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:10.999849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6ff29bf-1d39-4e64-8319-67465096aff0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:11.066919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ff29bf-1d39-4e64-8319-67465096aff0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:11.068426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6ff29bf-1d39-4e64-8319-67465096aff0 service glance] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:11.080463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 982/1945] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:10 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 254 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:11.387415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:11.389785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:11.467822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aece9724-dd1b-4c28-bed7-b8c08ca94aed admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:11.478666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:11.479350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:11.479801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:11.480303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] PATH_INFO: `/v3/projects/b7f9d8b6e0c742e1bf847f2eca04c196` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:11.482339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b7f9d8b6e0c742e1bf847f2eca04c196)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:11.484730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:11.644783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b7f9d8b6e0c742e1bf847f2eca04c196', 'request_id': 'req-09329c20-d114-44ef-b66f-5c9b9982c093'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:11.681231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b7f9d8b6e0c742e1bf847f2eca04c196', 'request_id': 'req-09329c20-d114-44ef-b66f-5c9b9982c093'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:11.775033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b7f9d8b6e0c742e1bf847f2eca04c196', 'request_id': 'req-09329c20-d114-44ef-b66f-5c9b9982c093'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:11.791580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] The token cache is being invalidate because project b7f9d8b6e0c742e1bf847f2eca04c196 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:11.791975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:11.792748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-09329c20-d114-44ef-b66f-5c9b9982c093 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:11.795865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 964/1946] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:11 2022] DELETE /identity/v3/projects/b7f9d8b6e0c742e1bf847f2eca04c196 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:11.803763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:11.804308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:11.804383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:11.827296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:11.829158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:11.945537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 983/1947] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:11.955580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:11.955755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:11.956008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:11.979927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:11.981233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:11.989793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 965/1948] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:12.002220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.014557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b7daba-b773-40c5-8e19-ab7cd845340e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.025784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.026456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.026793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.027078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.031198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:12.031198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.031198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:12.053184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 984/1949] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:37:12.053309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.074611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-429e5f4e-6465-4ac8-8e2f-698f0d9555cc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.081889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.082512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.082812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.083089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.083749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.086328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:12.208714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:12.231917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 966/1950] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:12 2022] POST /identity/v3/projects => generated 383 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:12.239317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.257694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27fd363d-5902-46a3-b422-0f89e84c3f3a admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.359669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.360405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.360746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.361080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.361782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.369886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:12.481586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 985/1951] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:12 2022] POST /identity/v3/users => generated 292 bytes in 241 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:12.482915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.495862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ffd15a5-9292-42fe-9c3f-6cd1dd695e14 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.506706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.507363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.507658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.507921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.508574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.510292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:12.521642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 967/1952] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:12 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:12.525763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.537939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9a460b-a514-4b65-a179-34b51b0fd3fa admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.570612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b04b51c0-bef9-4a92-b720-0398e0ad161e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.573030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04b51c0-bef9-4a92-b720-0398e0ad161e admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.573434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04b51c0-bef9-4a92-b720-0398e0ad161e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.573809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b04b51c0-bef9-4a92-b720-0398e0ad161e admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.575348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04b51c0-bef9-4a92-b720-0398e0ad161e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.577331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b04b51c0-bef9-4a92-b720-0398e0ad161e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:12.594959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 986/1953] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:12 2022] GET /identity/v3/roles => generated 1420 bytes in 70 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:12.595823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.607715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb53076b-f27b-4bc1-945b-24f9aee2c872 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.612535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.613374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.613778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.625644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] PATH_INFO: `/v3/projects/1ff0ec52888842eaa0da978008bc1b9a/users/b24facb59f4b41d1a53cbadb0cde8456/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.631006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1ff0ec52888842eaa0da978008bc1b9a, user_id=b24facb59f4b41d1a53cbadb0cde8456, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.633509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:12.725166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 968/1954] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:12 2022] PUT /identity/v3/projects/1ff0ec52888842eaa0da978008bc1b9a/users/b24facb59f4b41d1a53cbadb0cde8456/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:12.739770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.740870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.740870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.774281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] MFA Rules not processed for user `b24facb59f4b41d1a53cbadb0cde8456`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:12.775964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:12.867659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:12.867894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 987/1955] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:12 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:12.959872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19df2311-ffa4-4750-b2c4-3fbd005d7b55 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:12.965120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:12.967349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:12.967349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:12.967349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:12.967349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:12.968968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:13.071807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:13.120681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 969/1956] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:12 2022] POST /identity/v3/projects => generated 381 bytes in 254 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:13.124162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:13.138962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45e37203-cfde-4ff8-8447-a892acc39cec None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:13.265775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:13.266649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:13.266791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:13.267003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:13.267685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:13.269765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:13.424794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 988/1957] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:13 2022] POST /identity/v3/users => generated 297 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:13.427405 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:13.448898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5475a13a-29eb-4612-adee-366b6a5ab693 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:13.474442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:13.475478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:13.475972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:13.476443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:13.477473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:13.479530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:13.498282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 970/1958] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:13 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:13.499451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:13.514383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732eae25-d405-4d77-95cb-9e774381da64 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:13.537316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:13.539019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:13.539019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:13.539019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] PATH_INFO: `/v3/projects/e379a54e25714b35be8cdd12c57035d8/users/47c0137e03ac42edb9f2669dce99d199/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:13.557358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:13.564041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] RBAC: Authorizing `identity:create_grant(project_id=e379a54e25714b35be8cdd12c57035d8, user_id=47c0137e03ac42edb9f2669dce99d199, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:13.566368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:13.640448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 989/1959] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:13 2022] PUT /identity/v3/projects/e379a54e25714b35be8cdd12c57035d8/users/47c0137e03ac42edb9f2669dce99d199/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 139 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:13.642460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:13.654490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff3d6e8a-2801-4629-a936-dd961911ec2e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:13.724039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ad992f9-e046-447a-b853-cdc553d4c014 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:13.724760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad992f9-e046-447a-b853-cdc553d4c014 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:13.724986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad992f9-e046-447a-b853-cdc553d4c014 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:13.725366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ad992f9-e046-447a-b853-cdc553d4c014 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:13.726147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad992f9-e046-447a-b853-cdc553d4c014 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:13.728214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ad992f9-e046-447a-b853-cdc553d4c014 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:13.744356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 971/1960] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:13 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:13.745027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:13.755791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c7415c3-d1f3-44fd-aa63-28c5715d195c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:13.761154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:13.762480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:13.762834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:13.763115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] PATH_INFO: `/v3/projects/e379a54e25714b35be8cdd12c57035d8/users/47c0137e03ac42edb9f2669dce99d199/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:13.765549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] RBAC: Authorizing `identity:create_grant(project_id=e379a54e25714b35be8cdd12c57035d8, user_id=47c0137e03ac42edb9f2669dce99d199, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:13.768701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:13.782200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 990/1961] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:13 2022] PUT /identity/v3/projects/e379a54e25714b35be8cdd12c57035d8/users/47c0137e03ac42edb9f2669dce99d199/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 37 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:13.792632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:13.792942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:13.793222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:13.822578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] MFA Rules not processed for user `47c0137e03ac42edb9f2669dce99d199`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:13.823843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:13.914294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 972/1962] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:13 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:13.965228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:13.967539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:14.037931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:14.039723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60ace62-9c97-4242-8923-cbedb99049ff admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:14.046812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:14.047656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:14.048062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:14.048438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:14.067049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:14.067049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:14.074296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 991/1963] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:13 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 110 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:14.498114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:14.519516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fea6e61d-cf5f-4215-aa1d-b1062f89bb6a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:14.524601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40438caf-f509-4a07-9f60-34c98b87b097 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:14.525219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40438caf-f509-4a07-9f60-34c98b87b097 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:14.525493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40438caf-f509-4a07-9f60-34c98b87b097 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:14.525761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40438caf-f509-4a07-9f60-34c98b87b097 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:14.562564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:14.589155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:14.612312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40438caf-f509-4a07-9f60-34c98b87b097 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:14.613952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40438caf-f509-4a07-9f60-34c98b87b097 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:14.643908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 973/1964] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:14 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:15.260802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:15.573977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:15.597109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:16.001112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:16.429558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:16.911729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:17.208205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:17.437054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:17.695691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:17.880318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:18.180385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:18.454062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:18.527101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:18.528861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:18.594535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ff88f0-9f1a-484c-b1b6-127f81f66b0c service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:18.599236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:18.599856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:18.600128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:18.600397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] PATH_INFO: `/v3/users/b4fcf1c40e0c4a05a6622ef6d92fb452` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:18.601526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b4fcf1c40e0c4a05a6622ef6d92fb452)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:18.603558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:18.795828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:18.810047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b4fcf1c40e0c4a05a6622ef6d92fb452'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:18.810702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b4fcf1c40e0c4a05a6622ef6d92fb452'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:18.817472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b4fcf1c40e0c4a05a6622ef6d92fb452'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:18.818088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b4fcf1c40e0c4a05a6622ef6d92fb452'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:18.896066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b4fcf1c40e0c4a05a6622ef6d92fb452'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:18.914500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 992/1965] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:18 2022] DELETE /identity/v3/users/b4fcf1c40e0c4a05a6622ef6d92fb452 => generated 0 bytes in 388 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:18.921022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:18.921541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:18.922271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:18.944211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:18.945414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:19.005521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 974/1966] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:18 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:19.015385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:19.016881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:19.075979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-285ece65-137c-41ae-a7bf-3a7493e23370 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:19.079593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d74f09-a420-4937-8840-6dac1ded8b90 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:19.080199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d74f09-a420-4937-8840-6dac1ded8b90 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:19.080498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d74f09-a420-4937-8840-6dac1ded8b90 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:19.080789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d74f09-a420-4937-8840-6dac1ded8b90 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:19.090542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d74f09-a420-4937-8840-6dac1ded8b90 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:19.092059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d74f09-a420-4937-8840-6dac1ded8b90 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:19.101659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 993/1967] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:19.134158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=83,keystone:DELETE=10,keystone:INSERT=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:19.183829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:19.255270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:19.257290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:19.268424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5adb590-7663-4895-a4d5-b994ead4ff4c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:19.272893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:19.273516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:19.273790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:19.274064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] PATH_INFO: `/v3/users/b24facb59f4b41d1a53cbadb0cde8456` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:19.275193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b24facb59f4b41d1a53cbadb0cde8456)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:19.277610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:19.451896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:19.469893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b24facb59f4b41d1a53cbadb0cde8456'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.483864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b24facb59f4b41d1a53cbadb0cde8456'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.484628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b24facb59f4b41d1a53cbadb0cde8456'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.485982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b24facb59f4b41d1a53cbadb0cde8456'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.524283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:19.597212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b24facb59f4b41d1a53cbadb0cde8456'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.622091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 975/1968] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:19 2022] DELETE /identity/v3/users/b24facb59f4b41d1a53cbadb0cde8456 => generated 0 bytes in 367 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:19.626805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:19.627192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:19.627471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:19.653217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:19.654749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:19.656322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:19.658361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:19.727781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 994/1969] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:19 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:19.747640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:19.749296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:19.759513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a660b4ba-8147-4da8-ae66-ab4cffe01da0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:19.765927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:19.766591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:19.772228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:19.772228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] PATH_INFO: `/v3/projects/9da58a8167844d55826b167bfd25f960` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:19.772228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9da58a8167844d55826b167bfd25f960)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:19.772228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:19.815280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f54e377e-b808-4db7-b4f2-d9acddc32f84 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:19.818893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:19.822432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:19.822432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:19.822432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:19.829808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:19.831274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:19.841223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 995/1970] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:19 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 96 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:19.865945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:19.885937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9da58a8167844d55826b167bfd25f960', 'request_id': 'req-8ca677a6-7c54-4ffd-9466-a669324f71e3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.892079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9da58a8167844d55826b167bfd25f960', 'request_id': 'req-8ca677a6-7c54-4ffd-9466-a669324f71e3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.979729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:19.981460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9da58a8167844d55826b167bfd25f960', 'request_id': 'req-8ca677a6-7c54-4ffd-9466-a669324f71e3'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.995022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] The token cache is being invalidate because project 9da58a8167844d55826b167bfd25f960 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:19.995294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.996635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:19.998174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 976/1971] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:19 2022] DELETE /identity/v3/projects/9da58a8167844d55826b167bfd25f960 => generated 0 bytes in 343 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:20.002306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:20.012469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54e7d394-79fe-4e27-bdb8-f5d2c7809b93 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:20.073927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:20.075185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:20.075512 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:20.075780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] PATH_INFO: `/v3/users/297ed9c9f63a42fda82b0fd4a25e6eae` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:20.076917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] RBAC: Authorizing `identity:delete_user(user_id=297ed9c9f63a42fda82b0fd4a25e6eae)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:20.079066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:20.224080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=94,keystone:INSERT=5,keystone:DELETE=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:20.256227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '297ed9c9f63a42fda82b0fd4a25e6eae'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.256959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '297ed9c9f63a42fda82b0fd4a25e6eae'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.268499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:20.270211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:20.273356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '297ed9c9f63a42fda82b0fd4a25e6eae'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.274106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '297ed9c9f63a42fda82b0fd4a25e6eae'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.285880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca677a6-7c54-4ffd-9466-a669324f71e3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:20.291284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:20.291969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:20.292253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:20.292529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] PATH_INFO: `/v3/projects/1ff0ec52888842eaa0da978008bc1b9a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:20.294286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1ff0ec52888842eaa0da978008bc1b9a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:20.296220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:20.338223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '297ed9c9f63a42fda82b0fd4a25e6eae'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.359383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 996/1972] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:20 2022] DELETE /identity/v3/users/297ed9c9f63a42fda82b0fd4a25e6eae => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:20.438389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1ff0ec52888842eaa0da978008bc1b9a', 'request_id': 'req-32daff7f-9913-462d-96ea-0f85970d19a6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.444686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1ff0ec52888842eaa0da978008bc1b9a', 'request_id': 'req-32daff7f-9913-462d-96ea-0f85970d19a6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.453729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:20.516368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1ff0ec52888842eaa0da978008bc1b9a', 'request_id': 'req-32daff7f-9913-462d-96ea-0f85970d19a6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.529250 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] The token cache is being invalidate because project 1ff0ec52888842eaa0da978008bc1b9a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:20.529538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.530170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.531695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 977/1973] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:20 2022] DELETE /identity/v3/projects/1ff0ec52888842eaa0da978008bc1b9a => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:20.536126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:20.538093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:20.607341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5002939a-3365-4eee-86ea-815862cfce33 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:20.675190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:20.676253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:20.676253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:20.676508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] PATH_INFO: `/v3/users/47c0137e03ac42edb9f2669dce99d199` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:20.677507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] RBAC: Authorizing `identity:delete_user(user_id=47c0137e03ac42edb9f2669dce99d199)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:20.679472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:20.920087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:20.921596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:20.935483 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32daff7f-9913-462d-96ea-0f85970d19a6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:20.940362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:20.941041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:20.941323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:20.941599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] PATH_INFO: `/v3/projects/0a2a6cfbd0954ecb8aac28c00f3d39e4` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:20.942728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a2a6cfbd0954ecb8aac28c00f3d39e4)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:20.944755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:20.958474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '47c0137e03ac42edb9f2669dce99d199'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.959367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '47c0137e03ac42edb9f2669dce99d199'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.971431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '47c0137e03ac42edb9f2669dce99d199'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:20.972503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '47c0137e03ac42edb9f2669dce99d199'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.058368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a2a6cfbd0954ecb8aac28c00f3d39e4', 'request_id': 'req-e7286ae2-3483-4e5e-8275-18c557c80eee'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.065385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-637e9343-7894-4874-b21b-dec235e6e232 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '47c0137e03ac42edb9f2669dce99d199'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.065583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a2a6cfbd0954ecb8aac28c00f3d39e4', 'request_id': 'req-e7286ae2-3483-4e5e-8275-18c557c80eee'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.068005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:DELETE=9,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:21.086627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 997/1974] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:20 2022] DELETE /identity/v3/users/47c0137e03ac42edb9f2669dce99d199 => generated 0 bytes in 551 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:21.172257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a2a6cfbd0954ecb8aac28c00f3d39e4', 'request_id': 'req-e7286ae2-3483-4e5e-8275-18c557c80eee'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.190550 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] The token cache is being invalidate because project 0a2a6cfbd0954ecb8aac28c00f3d39e4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:21.190878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.192283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e7286ae2-3483-4e5e-8275-18c557c80eee admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.194256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 978/1975] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:20 2022] DELETE /identity/v3/projects/0a2a6cfbd0954ecb8aac28c00f3d39e4 => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:21.200536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.200851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.201129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.225311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:21.226562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.248976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:21.304273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 998/1976] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:21.313914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.314686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.315264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.339023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.340615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.361502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6561129-6bfb-44c7-a40d-908e84c9d903 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:21.370230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:21.371829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.372095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.372422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] PATH_INFO: `/v3/users/60766b0ad9f549ef8eb76064065fffd7` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.373588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:21.373751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=60766b0ad9f549ef8eb76064065fffd7)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:21.375100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.375974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:21.387435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 979/1977] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 72 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:21.389047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.400296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef332c4e-588a-4934-afa1-4790a280113b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:21.413221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:21.413848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.414127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.414561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.415273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:21.415580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:21.417226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:21.430220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 980/1978] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:37:21.434481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.445676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48f1af90-613e-40de-a234-98771b025f6d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:21.450914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:21.451539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.451823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.452099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.452739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:21.454698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:21.521139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:21.584024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 981/1979] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:21 2022] POST /identity/v3/projects => generated 383 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:21.588277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.628794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '60766b0ad9f549ef8eb76064065fffd7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.629827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '60766b0ad9f549ef8eb76064065fffd7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.640914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '60766b0ad9f549ef8eb76064065fffd7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.641595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '60766b0ad9f549ef8eb76064065fffd7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.674580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67dbafca-09f8-4369-a7eb-124ebd98cdea admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:21.679321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2b3e960-c568-4d9b-909f-0b57195f2844 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:21.679934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b3e960-c568-4d9b-909f-0b57195f2844 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.680218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b3e960-c568-4d9b-909f-0b57195f2844 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.680487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2b3e960-c568-4d9b-909f-0b57195f2844 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.681123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b3e960-c568-4d9b-909f-0b57195f2844 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:21.685994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2b3e960-c568-4d9b-909f-0b57195f2844 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:21.733938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '60766b0ad9f549ef8eb76064065fffd7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.755570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 999/1980] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:21 2022] DELETE /identity/v3/users/60766b0ad9f549ef8eb76064065fffd7 => generated 0 bytes in 417 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:21.756429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.757797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.775797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 982/1981] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:21 2022] POST /identity/v3/users => generated 292 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:21.777961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.778251 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.778519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.799319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:21.805265 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.847558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e94bf4-02d8-418c-b811-b1e29dc423d6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:21.855791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:21.856533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.856806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.857080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] PATH_INFO: `/v3/projects/e379a54e25714b35be8cdd12c57035d8` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.858513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e379a54e25714b35be8cdd12c57035d8)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:21.860491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:21.861989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:21.877693 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 983/1982] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:21 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:21.878749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.880147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.898418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:21.898589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244d0ab0-286b-4473-aace-0bfdf9804af1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:21.898589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:21.899270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:21.899547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:21.899824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:21.900447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:21.902048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:21.913340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 984/1983] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:21 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:21.914146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:21.915752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:21.971993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e379a54e25714b35be8cdd12c57035d8', 'request_id': 'req-38806a1a-ae97-4341-b4e5-e798e72f793f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:21.980076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0db8f2-7032-4dda-93e4-7aa587266e5d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.031817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79601b90-6216-4a47-ae7d-93872fad1e82 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.032429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79601b90-6216-4a47-ae7d-93872fad1e82 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.032705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79601b90-6216-4a47-ae7d-93872fad1e82 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.032980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79601b90-6216-4a47-ae7d-93872fad1e82 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.036519 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e379a54e25714b35be8cdd12c57035d8', 'request_id': 'req-38806a1a-ae97-4341-b4e5-e798e72f793f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.044570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e379a54e25714b35be8cdd12c57035d8', 'request_id': 'req-38806a1a-ae97-4341-b4e5-e798e72f793f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.060500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] The token cache is being invalidate because project e379a54e25714b35be8cdd12c57035d8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:22.060874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.061699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.063625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1000/1984] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:21 2022] DELETE /identity/v3/projects/e379a54e25714b35be8cdd12c57035d8 => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:22.064619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.068143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.117034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79601b90-6216-4a47-ae7d-93872fad1e82 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.118491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79601b90-6216-4a47-ae7d-93872fad1e82 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.129408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 985/1985] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:21 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:22.130647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.130962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.131235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.140083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38806a1a-ae97-4341-b4e5-e798e72f793f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.146432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f5d9f1-389b-4736-ac50-b42abf912dc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.146432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5d9f1-389b-4736-ac50-b42abf912dc5 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.146432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5d9f1-389b-4736-ac50-b42abf912dc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.146432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f5d9f1-389b-4736-ac50-b42abf912dc5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.146932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5d9f1-389b-4736-ac50-b42abf912dc5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.148477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f5d9f1-389b-4736-ac50-b42abf912dc5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.153425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:22.154645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.172202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 986/1987] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:22.175202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1001/1987] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:22 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:22.175328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.175420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.175420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.175420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.179657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f70cb226-5739-42ad-ba93-7c47d3079167 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.186244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8c65c7e-68ec-460b-9adc-f98b955940c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.187068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c65c7e-68ec-460b-9adc-f98b955940c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.187347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c65c7e-68ec-460b-9adc-f98b955940c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.187619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8c65c7e-68ec-460b-9adc-f98b955940c6 admin admin] PATH_INFO: `/v3/projects/db05a9796b1841abac93bfbabda8728f/users/a1bd2fda86894b7b95fa71e06c93867e/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.197908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:22.200035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.204997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c65c7e-68ec-460b-9adc-f98b955940c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db05a9796b1841abac93bfbabda8728f, user_id=a1bd2fda86894b7b95fa71e06c93867e, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.208608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8c65c7e-68ec-460b-9adc-f98b955940c6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.211524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1002/1988] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:22.216296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.225318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b49a438-7b27-4a4c-b159-af0c9e7e8a64 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.230400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.231016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.232421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.232421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.232421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:22.232657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.234077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.246634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1003/1989] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:22.251690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.264265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f84080-e807-4369-b554-1934bc26f200 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.269048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.269679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.269954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.270222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.270884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.272788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.292065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 987/1990] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:22 2022] PUT /identity/v3/projects/db05a9796b1841abac93bfbabda8728f/users/a1bd2fda86894b7b95fa71e06c93867e/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:22.303457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.303769 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.304043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.320965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:22.344104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] MFA Rules not processed for user `a1bd2fda86894b7b95fa71e06c93867e`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:22.345365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.382473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1004/1991] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:22 2022] POST /identity/v3/projects => generated 377 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:22.393103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.404817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:22.441315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 988/1992] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:22 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:22.454810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.456458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.473727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1158bd1b-27cd-4627-a06b-72613baeb3ca admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.480242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.481177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.481477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.481745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.482400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.484925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.516147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c77ccc8a-aa94-47ba-b7db-011c2281d5b5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.564457 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.565091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.565367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.565635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.618250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1005/1993] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:22 2022] POST /identity/v3/users => generated 289 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:22.619295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.622902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.633465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.635020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.641428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5b6c3db-2d60-4c9f-aaa5-04a4549ea889 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.645985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 989/1994] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:22 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:22.647125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.656547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.657207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.657478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.658678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] PATH_INFO: `/v3/projects/78b55416bb584f8aa9992d308e1b1e1b` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.661791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] RBAC: Authorizing `identity:delete_project(project_id=78b55416bb584f8aa9992d308e1b1e1b)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.663598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c64fcca-b207-423c-8f75-a83ffe1a3254 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.664104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.671839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.673263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.673544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.673814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.674459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.676209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.687766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 990/1995] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:22 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:22.688626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.698524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3616159-cb2b-475c-8ca2-8ebb7ce4a179 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.703382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.704004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.704346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.704627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] PATH_INFO: `/v3/users/a1bd2fda86894b7b95fa71e06c93867e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.705787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a1bd2fda86894b7b95fa71e06c93867e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:22.707755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:22.722478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:22.785518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '78b55416bb584f8aa9992d308e1b1e1b', 'request_id': 'req-83055342-0948-44e7-b1b5-00cbf5cc0b5e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.888995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '78b55416bb584f8aa9992d308e1b1e1b', 'request_id': 'req-83055342-0948-44e7-b1b5-00cbf5cc0b5e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.896996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '78b55416bb584f8aa9992d308e1b1e1b', 'request_id': 'req-83055342-0948-44e7-b1b5-00cbf5cc0b5e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.908400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] The token cache is being invalidate because project 78b55416bb584f8aa9992d308e1b1e1b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:22.908612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.909293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.910876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1006/1996] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:22 2022] DELETE /identity/v3/projects/78b55416bb584f8aa9992d308e1b1e1b => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:22.912043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:22.914106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:22.972843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a1bd2fda86894b7b95fa71e06c93867e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.981512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a1bd2fda86894b7b95fa71e06c93867e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.982993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a1bd2fda86894b7b95fa71e06c93867e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.983783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a1bd2fda86894b7b95fa71e06c93867e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:22.991896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83055342-0948-44e7-b1b5-00cbf5cc0b5e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:22.996705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:22.997414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:22.997694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:22.997994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:22.998614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.000258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:23.010659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1007/1997] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:22 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:23.011614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.013460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.026384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0623247-3873-4875-8686-8bdba5acddd7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.032273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:23.032906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.033234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.033478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] PATH_INFO: `/v3/users/2554247492024c7da62c0aae89b7a70b` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.034606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2554247492024c7da62c0aae89b7a70b)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.036851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:23.046274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a1bd2fda86894b7b95fa71e06c93867e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:23.065648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 991/1998] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:22 2022] DELETE /identity/v3/users/a1bd2fda86894b7b95fa71e06c93867e => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:23.066614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.067970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.135555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c58619-5e62-4a59-8133-de6e07c27a40 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.139963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f652fe4-8f8d-4537-b4d8-855d227f31d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:23.140629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f652fe4-8f8d-4537-b4d8-855d227f31d4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.140992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f652fe4-8f8d-4537-b4d8-855d227f31d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.142156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f652fe4-8f8d-4537-b4d8-855d227f31d4 admin admin] PATH_INFO: `/v3/projects/ee72d024fa844e0fb199018e5a52887e/users/1bb251c2304d477dbcc13296c34edb3c/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.156748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f652fe4-8f8d-4537-b4d8-855d227f31d4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee72d024fa844e0fb199018e5a52887e, user_id=1bb251c2304d477dbcc13296c34edb3c, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.159654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f652fe4-8f8d-4537-b4d8-855d227f31d4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:23.209702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2554247492024c7da62c0aae89b7a70b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:23.210363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2554247492024c7da62c0aae89b7a70b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:23.218196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2554247492024c7da62c0aae89b7a70b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:23.218897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2554247492024c7da62c0aae89b7a70b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:23.224181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:23.229311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 992/1999] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:23 2022] PUT /identity/v3/projects/ee72d024fa844e0fb199018e5a52887e/users/1bb251c2304d477dbcc13296c34edb3c/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:23.231376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc687656-3f24-447d-96d3-cb6ffd38276c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.231660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc687656-3f24-447d-96d3-cb6ffd38276c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.231929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc687656-3f24-447d-96d3-cb6ffd38276c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.250924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bc687656-3f24-447d-96d3-cb6ffd38276c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:23.252160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bc687656-3f24-447d-96d3-cb6ffd38276c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.299683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2554247492024c7da62c0aae89b7a70b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:23.316779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 993/2000] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:23.318027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.318314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.318591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.321379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1008/2001] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:23 2022] DELETE /identity/v3/users/2554247492024c7da62c0aae89b7a70b => generated 0 bytes in 307 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:23.326830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.334032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.358689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] MFA Rules not processed for user `1bb251c2304d477dbcc13296c34edb3c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:23.359976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.402852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f287d02-84ec-4141-b087-990d34cdbb69 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.407201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:23.408034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.408294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.408630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.441432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 994/2002] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:23.446158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.447647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.477618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.479120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:23.488178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:23.491892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1009/2003] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 166 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:23.525537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5981eb22-e740-4252-913c-f98bdb181c2e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.531242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:23.531882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.532235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.532567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.533178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.534236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:23.535228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:23.682908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 995/2004] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:23 2022] POST /identity/v3/projects => generated 375 bytes in 233 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:23.683932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.698758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c756081-d4fb-4d8d-bd7f-4f79e97bc66c service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.756539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:23.757357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.757636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.757981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.758604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.762885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:23.946130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.948257 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1010/2005] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:23 2022] POST /identity/v3/users => generated 288 bytes in 262 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:23.957187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:23.964331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:23.967381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30662370-fd4b-4e83-b81d-77332f87bb53 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.977487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:23.978642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:23.979152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:23.979695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:23.980014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-897f2cc0-18dc-4529-a9d5-0458c5297dd1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:23.981980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:23.984434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.002443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 996/2006] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:23 2022] GET /identity/v3/roles => generated 1420 bytes in 57 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:24.004110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.016171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:24.017845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.017989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c29c132-1936-4031-8541-ca8d6566529a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.019047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.019395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.019643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] PATH_INFO: `/v3/projects/9545e05876d14ef08ab58ebd18bf3103` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.020825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] RBAC: Authorizing `identity:delete_project(project_id=9545e05876d14ef08ab58ebd18bf3103)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.022766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.026732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.027398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.027775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.028128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.028691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.030323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.042098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 997/2007] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:24 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:24.046645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.056952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff7822f-8af1-4854-83fb-4d7f6f1aa45d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.063341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-501d6799-9754-4109-b1a7-ffc251a4754e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.065557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:24.066245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d6799-9754-4109-b1a7-ffc251a4754e admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.066697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d6799-9754-4109-b1a7-ffc251a4754e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.067846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-501d6799-9754-4109-b1a7-ffc251a4754e admin admin] PATH_INFO: `/v3/projects/d576c830819f484fb2c13bf21ce92017/users/4085f888392d46188b32e0c7ba2b2c34/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.089812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d6799-9754-4109-b1a7-ffc251a4754e admin admin] RBAC: Authorizing `identity:create_grant(project_id=d576c830819f484fb2c13bf21ce92017, user_id=4085f888392d46188b32e0c7ba2b2c34, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.096906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-501d6799-9754-4109-b1a7-ffc251a4754e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.189968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 998/2008] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:24 2022] PUT /identity/v3/projects/d576c830819f484fb2c13bf21ce92017/users/4085f888392d46188b32e0c7ba2b2c34/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:24.203069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e349218-66ca-40a5-b804-d1261296a135 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.203386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e349218-66ca-40a5-b804-d1261296a135 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.203635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e349218-66ca-40a5-b804-d1261296a135 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.205517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9545e05876d14ef08ab58ebd18bf3103', 'request_id': 'req-c0572a07-1a74-49f5-bfcc-82f54be7b88f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.245401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2e349218-66ca-40a5-b804-d1261296a135 None None] MFA Rules not processed for user `4085f888392d46188b32e0c7ba2b2c34`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:24.248378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2e349218-66ca-40a5-b804-d1261296a135 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.280572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9545e05876d14ef08ab58ebd18bf3103', 'request_id': 'req-c0572a07-1a74-49f5-bfcc-82f54be7b88f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.291358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9545e05876d14ef08ab58ebd18bf3103', 'request_id': 'req-c0572a07-1a74-49f5-bfcc-82f54be7b88f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.295281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:24.307138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] The token cache is being invalidate because project 9545e05876d14ef08ab58ebd18bf3103 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:24.307475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.308708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.310548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1011/2009] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:23 2022] DELETE /identity/v3/projects/9545e05876d14ef08ab58ebd18bf3103 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:24.314993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.314993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.353275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 999/2010] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:24.353275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.353275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.353275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.372367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:24.374360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.389659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1000/2011] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:24.389659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.389964 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0572a07-1a74-49f5-bfcc-82f54be7b88f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.390059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.394422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.395266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.398444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.398444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] PATH_INFO: `/v3/projects/db05a9796b1841abac93bfbabda8728f` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.398444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=db05a9796b1841abac93bfbabda8728f)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.399922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e602b9be-f6ce-4874-8b63-e8023fa5fd97 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.400035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.405212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2aafee9-f154-4cc3-bd34-effcff095c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.405826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2aafee9-f154-4cc3-bd34-effcff095c37 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.406131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2aafee9-f154-4cc3-bd34-effcff095c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.406455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2aafee9-f154-4cc3-bd34-effcff095c37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.407124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2aafee9-f154-4cc3-bd34-effcff095c37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.409028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2aafee9-f154-4cc3-bd34-effcff095c37 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.413715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:24.524421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1001/2012] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/projects => generated 375 bytes in 136 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:24.524421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.524421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.524421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.537185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db05a9796b1841abac93bfbabda8728f', 'request_id': 'req-b4b775de-f042-4854-b20c-20e5aef362a8'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.540521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:24.546393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:24.547666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.601959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db05a9796b1841abac93bfbabda8728f', 'request_id': 'req-b4b775de-f042-4854-b20c-20e5aef362a8'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.609592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db05a9796b1841abac93bfbabda8728f', 'request_id': 'req-b4b775de-f042-4854-b20c-20e5aef362a8'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.613754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1002/2013] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:24.614793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.622398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] The token cache is being invalidate because project db05a9796b1841abac93bfbabda8728f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:24.622809 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.623555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:24.626398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1012/2014] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:24 2022] DELETE /identity/v3/projects/db05a9796b1841abac93bfbabda8728f => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:24.627271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.628870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.630852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b8eb0e-f36c-4afa-a74a-aa292601112e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.637532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b65931-10e6-47d7-b834-80ee73dc9e0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.638445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b65931-10e6-47d7-b834-80ee73dc9e0c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.638907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b65931-10e6-47d7-b834-80ee73dc9e0c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.639308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b65931-10e6-47d7-b834-80ee73dc9e0c admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.640165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b65931-10e6-47d7-b834-80ee73dc9e0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.643004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b65931-10e6-47d7-b834-80ee73dc9e0c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.702151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b775de-f042-4854-b20c-20e5aef362a8 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.706725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.707351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.707630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.707918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.708562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:24.708862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.710442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e2dfb88-8538-440a-9da6-63f24325ddc8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.721888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1013/2015] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:24.723091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.723368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.723997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.728638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:INSERT=3,keystone:SELECT=87,keystone:DELETE=10 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:24.744950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:24.746741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.756845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1014/2016] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:24.757685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.768581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e325a02-f7ee-4c6f-9e9f-fa32eb1f172e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.773163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.773767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.774043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.774314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.775029 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.776881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.791289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1003/2017] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/users => generated 294 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:24.800255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.800255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.800255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.814409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:24.815758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.824259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1004/2018] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:24.825081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.829214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:24.843563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c377781-77be-4d59-9372-6bca248b0a02 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.849200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.849830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.850112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.850390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.851102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.852758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.865482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1005/2019] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:24 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:24.866622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.878583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e764e71-1f49-4666-8aac-9d7ce1085d88 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.883502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.884121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.884408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.884684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.885363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:24.885794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.888012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.900651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1006/2020] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:24.901581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.904412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1015/2021] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/projects => generated 381 bytes in 147 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:24.905406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:24.914428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-702a36ef-1909-4747-90dd-b36df6111bec admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.973058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd1bf742-0b98-4603-8c75-e751445c592f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:24.978238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.978448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:24.978942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.979071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:24.979234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.979349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:24.979512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] PATH_INFO: `/v3/projects/dbc7e83dd3d3429aaa953500ef3ae337/users/9d9458049a384442806df3243f9a50fe/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.979604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:24.980861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.983493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:24.996541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc7e83dd3d3429aaa953500ef3ae337, user_id=9d9458049a384442806df3243f9a50fe, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:24.999554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.092834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1016/2022] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:24 2022] POST /identity/v3/users => generated 291 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:25.093870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.101552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:25.104183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f016811-2bbe-45b6-b813-7288f980d9a2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.109149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1007/2023] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:24 2022] PUT /identity/v3/projects/dbc7e83dd3d3429aaa953500ef3ae337/users/9d9458049a384442806df3243f9a50fe/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:25.110401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.180402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.181018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.181294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.181562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.183762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.184348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.189430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70bf5982-011a-4bca-a84f-9fc1fa255ca8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.194486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.195229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.195497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.195770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.196413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.198026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.209100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1008/2024] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 100 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:25.210010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.221345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b447c3e-eea9-4256-ad67-c2bd0af3825a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.226579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.227248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.227541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.227801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.228440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.230075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.241083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1009/2025] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:25.244855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.251519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da9234d1-9a0a-454b-a03f-de456babd9b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.256190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.256742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.257014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.257280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.257901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.259516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.272044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1017/2026] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:25 2022] POST /identity/v3/projects => generated 365 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:25.272196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1010/2027] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:25.272295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.279902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.290668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8615c1-e601-4ec5-b839-069905cab647 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.321800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:25.366507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73a45a-2602-47ea-bb59-089086e91540 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.367269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73a45a-2602-47ea-bb59-089086e91540 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.367620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73a45a-2602-47ea-bb59-089086e91540 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.367962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73a45a-2602-47ea-bb59-089086e91540 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.368191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d524fd-b2cc-449a-8c0d-9fbf991703f3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.369150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e73a45a-2602-47ea-bb59-089086e91540 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.371052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:25.371411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e73a45a-2602-47ea-bb59-089086e91540 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.375550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.375941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.376225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.376503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] PATH_INFO: `/v3/projects/dbc7e83dd3d3429aaa953500ef3ae337/users/9d9458049a384442806df3243f9a50fe/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.378471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] RBAC: Authorizing `identity:create_grant(project_id=dbc7e83dd3d3429aaa953500ef3ae337, user_id=9d9458049a384442806df3243f9a50fe, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.381262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.397409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1018/2028] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:25 2022] PUT /identity/v3/projects/dbc7e83dd3d3429aaa953500ef3ae337/users/9d9458049a384442806df3243f9a50fe/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:25.397409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.459306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1011/2029] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:25 2022] POST /identity/v3/users => generated 283 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:25.459306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.459306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.459306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.477841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87597b97-b7d8-4945-9e48-694663648bdf admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.486932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.487594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.487874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.488147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] PATH_INFO: `/v3/projects/4c1b29b0f86642a48aeeb68393f7d297/users/8af176c539774adaa8a06c8c3c797a68/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.489656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] MFA Rules not processed for user `9d9458049a384442806df3243f9a50fe`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:25.491038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:25.504572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4c1b29b0f86642a48aeeb68393f7d297, user_id=8af176c539774adaa8a06c8c3c797a68, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.507437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.562869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1012/2030] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:25 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:25.563725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.575907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07e6715-81ca-4573-8c34-3bf48047ab32 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.580539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.581160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.581437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.581710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.582340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.583964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.594920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1013/2031] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:25.595821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.597717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:25.619982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1019/2032] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:25 2022] PUT /identity/v3/projects/4c1b29b0f86642a48aeeb68393f7d297/users/8af176c539774adaa8a06c8c3c797a68/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:25.620860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.672362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b3c754b-1325-4e2a-ab65-2697d7f9920e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.696327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d340fb4-7de2-485c-a618-7eed19ea51b0 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.701856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b37dce-96b1-4092-adeb-63bd4d5820c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.702465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b37dce-96b1-4092-adeb-63bd4d5820c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.702998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b37dce-96b1-4092-adeb-63bd4d5820c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.703086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b37dce-96b1-4092-adeb-63bd4d5820c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.703815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b37dce-96b1-4092-adeb-63bd4d5820c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.705322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6b37dce-96b1-4092-adeb-63bd4d5820c2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.716081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1020/2033] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:25 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:25.717452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.717670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.717946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.723878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.724489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.724765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.725033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.726996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:25.729557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:25.750834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] MFA Rules not processed for user `8af176c539774adaa8a06c8c3c797a68`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:25.752498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:25.803629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.807629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.835764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1014/2034] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 221 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:25.835764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.835764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7187de8-b8f5-4ad1-b54b-d77e5d0a7dbd service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.836402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1021/2035] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:25 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:25.839299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:25.840800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8958d9d5-75bc-41a6-bc88-1e2370c83495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.841828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958d9d5-75bc-41a6-bc88-1e2370c83495 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.842311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958d9d5-75bc-41a6-bc88-1e2370c83495 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.842811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8958d9d5-75bc-41a6-bc88-1e2370c83495 admin admin] PATH_INFO: `/v3/projects/b4ef72b5c4db448293a748cdeace41d0/users/379c7b5e94c842b3b757450b345de820/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.853866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b62ade4-37f2-4ed8-85c8-dee3757707bc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:25.860058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:25.860764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:25.861444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:25.861722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:25.862368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.864372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.880478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8958d9d5-75bc-41a6-bc88-1e2370c83495 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b4ef72b5c4db448293a748cdeace41d0, user_id=379c7b5e94c842b3b757450b345de820, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:25.887578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8958d9d5-75bc-41a6-bc88-1e2370c83495 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:25.992272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1022/2036] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:25 2022] POST /identity/v3/projects => generated 381 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:25.996730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.014629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1015/2037] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:25 2022] PUT /identity/v3/projects/b4ef72b5c4db448293a748cdeace41d0/users/379c7b5e94c842b3b757450b345de820/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:26.025016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.025320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.025584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.055313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] MFA Rules not processed for user `379c7b5e94c842b3b757450b345de820`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:26.056537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:26.059828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b371d01-4285-48af-8673-c9f3914f80d3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.130723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.130907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.131075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.131343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.131968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.133950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.139313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1016/2038] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:26 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:26.147022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.148459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:26.206566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c142f0d1-82b1-4d41-b997-4a46924df49c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.210283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.210921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.211200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.211471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.220022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.221438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.230220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1017/2039] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:26 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:26.287717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1023/2040] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:25 2022] POST /identity/v3/users => generated 297 bytes in 292 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:26.292423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.311593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-701dfecb-c0a1-4ec4-9d22-a2a337960b4d service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.316351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.316960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.317238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.317531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.318163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.319853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.330402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1018/2041] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:26.335221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.347196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83a74938-a3f9-44d0-bb40-24beefa76dae admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.351886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.352552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.352873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.353081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] PATH_INFO: `/v3/projects/aadd31f6a9ab462ba1a4c806d86bc6c1/users/9245df73af894e87aaaf3849407b1727/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.372501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aadd31f6a9ab462ba1a4c806d86bc6c1, user_id=9245df73af894e87aaaf3849407b1727, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.375439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.452125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1024/2042] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:26 2022] PUT /identity/v3/projects/aadd31f6a9ab462ba1a4c806d86bc6c1/users/9245df73af894e87aaaf3849407b1727/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:26.456389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.467451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c6f01c-f3e6-4349-b747-29d4e076362b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.535468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de45086f-e1de-45f0-bb0c-f1587f953f38 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.536334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de45086f-e1de-45f0-bb0c-f1587f953f38 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.536620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de45086f-e1de-45f0-bb0c-f1587f953f38 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.536909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de45086f-e1de-45f0-bb0c-f1587f953f38 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.537530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de45086f-e1de-45f0-bb0c-f1587f953f38 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.541216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de45086f-e1de-45f0-bb0c-f1587f953f38 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.551979 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1019/2043] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:26 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:26.556254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.567198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0907166c-4f16-4da4-8fcb-e692879708c5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.571708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.572409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.572647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.572917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] PATH_INFO: `/v3/projects/aadd31f6a9ab462ba1a4c806d86bc6c1/users/9245df73af894e87aaaf3849407b1727/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.575073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aadd31f6a9ab462ba1a4c806d86bc6c1, user_id=9245df73af894e87aaaf3849407b1727, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.577689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.588132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1025/2044] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:26 2022] PUT /identity/v3/projects/aadd31f6a9ab462ba1a4c806d86bc6c1/users/9245df73af894e87aaaf3849407b1727/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:26.600242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.600548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.600830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.632364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] MFA Rules not processed for user `9245df73af894e87aaaf3849407b1727`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:26.633602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:26.723720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1020/2045] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:26 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:26.763639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:26.824594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab32ab42-3977-459a-9bc8-462a2074f677 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:26.828441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44ff54ab-fafc-4bf1-a902-5ba7fba19c2a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:26.829168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ff54ab-fafc-4bf1-a902-5ba7fba19c2a service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:26.829558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ff54ab-fafc-4bf1-a902-5ba7fba19c2a service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:26.829993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44ff54ab-fafc-4bf1-a902-5ba7fba19c2a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:26.841421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ff54ab-fafc-4bf1-a902-5ba7fba19c2a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:26.846910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44ff54ab-fafc-4bf1-a902-5ba7fba19c2a service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:26.854011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1026/2046] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:26 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:26.899919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:27.486083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:27.549517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a2c0b17-8c6b-4548-bae8-f25a6599197a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:27.554271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:27.554953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:27.555230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:27.555499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] PATH_INFO: `/v3/users/8af176c539774adaa8a06c8c3c797a68` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:27.556797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8af176c539774adaa8a06c8c3c797a68)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:27.559378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:27.677026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:27.795868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8af176c539774adaa8a06c8c3c797a68'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:27.802968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8af176c539774adaa8a06c8c3c797a68'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:27.803650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8af176c539774adaa8a06c8c3c797a68'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:27.804279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8af176c539774adaa8a06c8c3c797a68'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:27.916451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8af176c539774adaa8a06c8c3c797a68'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:27.940084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1021/2047] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:27 2022] DELETE /identity/v3/users/8af176c539774adaa8a06c8c3c797a68 => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:27.961652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:27.961652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:27.961652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:27.963904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:27.965154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:28.033102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1027/2048] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:27 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:28.047911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:28.052557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:28.064393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:28.066201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:28.126938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70cce168-c9cc-43ff-bb96-eae1913844e1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:28.128522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffab4dc3-551f-4239-b802-ae13b86cbe36 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:28.131023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:28.131766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:28.132315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:28.132728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:28.132887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:28.133623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:28.133960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:28.134465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:28.144736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:28.147795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:28.149421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:28.164746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1022/2049] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:28.267907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:28.269549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:28.283207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1028/2050] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:28 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 219 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:28.810918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:28.813097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:28.835020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3692d2fa-7537-462c-a29b-53dfcdfdf172 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:28.844752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:28.845993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:28.846307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:28.846992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] PATH_INFO: `/v3/projects/4c1b29b0f86642a48aeeb68393f7d297` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:28.852396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4c1b29b0f86642a48aeeb68393f7d297)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:28.855981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:28.902806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:29.003259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4c1b29b0f86642a48aeeb68393f7d297', 'request_id': 'req-f7b36274-3beb-4c49-8750-11c4fb68b817'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.008789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4c1b29b0f86642a48aeeb68393f7d297', 'request_id': 'req-f7b36274-3beb-4c49-8750-11c4fb68b817'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.116699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4c1b29b0f86642a48aeeb68393f7d297', 'request_id': 'req-f7b36274-3beb-4c49-8750-11c4fb68b817'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.133428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] The token cache is being invalidate because project 4c1b29b0f86642a48aeeb68393f7d297 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:29.133849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.134595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.137023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1023/2051] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:28 2022] DELETE /identity/v3/projects/4c1b29b0f86642a48aeeb68393f7d297 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:29.141196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:29.153898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad67af40-6c7d-4903-8eff-53051a1a39ec service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:29.218202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:29.218921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:29.219352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:29.219579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] PATH_INFO: `/v3/users/9245df73af894e87aaaf3849407b1727` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:29.220755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9245df73af894e87aaaf3849407b1727)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:29.224766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:29.469492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9245df73af894e87aaaf3849407b1727'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.471176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9245df73af894e87aaaf3849407b1727'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.479635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9245df73af894e87aaaf3849407b1727'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.481893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9245df73af894e87aaaf3849407b1727'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.574498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9245df73af894e87aaaf3849407b1727'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:29.596954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1029/2052] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:29 2022] DELETE /identity/v3/users/9245df73af894e87aaaf3849407b1727 => generated 0 bytes in 456 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:29.786020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:30.135677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:30.151823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:30.169752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b36274-3beb-4c49-8750-11c4fb68b817 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:30.226512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:30.235381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:30.243771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.243771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.243771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] PATH_INFO: `/v3/projects/aadd31f6a9ab462ba1a4c806d86bc6c1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.243771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aadd31f6a9ab462ba1a4c806d86bc6c1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:30.243771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:30.244211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:30.244211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:30.309939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a7f541-49df-40e2-970d-558a8da3723b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:30.313570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:30.314173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.314451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.314724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.316637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:30.366386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aadd31f6a9ab462ba1a4c806d86bc6c1', 'request_id': 'req-443b4b64-f03e-4cd8-883c-b6d53d9f5143'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:30.373357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aadd31f6a9ab462ba1a4c806d86bc6c1', 'request_id': 'req-443b4b64-f03e-4cd8-883c-b6d53d9f5143'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:30.385793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:30.387940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0d31455-7e33-461f-a7df-ae721dd764e8 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:30.398882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1030/2053] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:30 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 158 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:30.471428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aadd31f6a9ab462ba1a4c806d86bc6c1', 'request_id': 'req-443b4b64-f03e-4cd8-883c-b6d53d9f5143'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:30.484150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] The token cache is being invalidate because project aadd31f6a9ab462ba1a4c806d86bc6c1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:30.484446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:30.485116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-443b4b64-f03e-4cd8-883c-b6d53d9f5143 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:30.486616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1024/2054] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:30 2022] DELETE /identity/v3/projects/aadd31f6a9ab462ba1a4c806d86bc6c1 => generated 0 bytes in 352 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:30.492577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.492577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.492577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.513222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:30.515575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:30.590206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1031/2055] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:30.594000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.599730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.599730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.622642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:30.622642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:30.629889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1025/2056] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:30.634046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:30.645394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eae0482d-4768-41d5-a8db-285bce0dc5ee None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:30.650253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:30.650890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.651173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.651447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.652299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:30.652714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:30.654333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:30.666695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1032/2057] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:30 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:30.672798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:30.682540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5ea79c-9216-4153-b1cb-56eeab98d056 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:30.689555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:30.690169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.690444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.690713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.691394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:30.693372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:30.823983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1026/2058] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:30 2022] POST /identity/v3/projects => generated 361 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:30.825158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:30.835391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72fc06d8-9cd5-4053-8480-68ff27ad9131 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:30.896097 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:30.896711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:30.896987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:30.897256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:30.898046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:30.900010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:30.984336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1033/2059] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:30 2022] POST /identity/v3/users => generated 281 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:30.988439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:30.999386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f605cd7a-9c63-40a9-902d-6a67cb958719 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:31.004072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:31.004712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.004990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.005285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.005936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:31.007541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:31.017542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1027/2060] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:30 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:31.021422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:31.031614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-750e8c31-fc3b-41f6-9240-9e631d0dc165 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:31.036992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45155983-bdda-4469-baa4-c19643be3003 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:31.037598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45155983-bdda-4469-baa4-c19643be3003 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.037880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45155983-bdda-4469-baa4-c19643be3003 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.038212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45155983-bdda-4469-baa4-c19643be3003 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.038880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45155983-bdda-4469-baa4-c19643be3003 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:31.040552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45155983-bdda-4469-baa4-c19643be3003 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:31.052265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1034/2061] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:31 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:31.060386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:31.070392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f531d489-b397-4d72-be83-0d297fd07600 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:31.077037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:31.077812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.078154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.078433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] PATH_INFO: `/v3/projects/817923d7c2464263969257ed51cc7607/users/4d2e379bb3894a3db93feb28c95a1367/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.112697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] RBAC: Authorizing `identity:create_grant(project_id=817923d7c2464263969257ed51cc7607, user_id=4d2e379bb3894a3db93feb28c95a1367, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:31.117498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:31.240905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1028/2062] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:31 2022] PUT /identity/v3/projects/817923d7c2464263969257ed51cc7607/users/4d2e379bb3894a3db93feb28c95a1367/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:31.252194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610e71d2-2594-41af-8955-706edb03a16d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.252682 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610e71d2-2594-41af-8955-706edb03a16d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.252821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-610e71d2-2594-41af-8955-706edb03a16d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.288556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-610e71d2-2594-41af-8955-706edb03a16d None None] MFA Rules not processed for user `4d2e379bb3894a3db93feb28c95a1367`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:31.289841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-610e71d2-2594-41af-8955-706edb03a16d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:31.363657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1035/2063] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:31 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:31.394017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:31.403759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:31.466641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9fb88be-8068-47a4-bd4a-5c891fa73058 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:31.470289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b71d27ef-b4b4-4bc2-8dcb-21a6ed2dfa7a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:31.470920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71d27ef-b4b4-4bc2-8dcb-21a6ed2dfa7a service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.471197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71d27ef-b4b4-4bc2-8dcb-21a6ed2dfa7a service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.471466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b71d27ef-b4b4-4bc2-8dcb-21a6ed2dfa7a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.480671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71d27ef-b4b4-4bc2-8dcb-21a6ed2dfa7a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:31.482137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b71d27ef-b4b4-4bc2-8dcb-21a6ed2dfa7a service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:31.491142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1029/2064] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:31 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:31.507117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610e71d2-2594-41af-8955-706edb03a16d None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:31.569391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-610e71d2-2594-41af-8955-706edb03a16d None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:31.574444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:31.575172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.575430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.575701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] PATH_INFO: `/v3/users/4d2e379bb3894a3db93feb28c95a1367` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.576815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4d2e379bb3894a3db93feb28c95a1367)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:31.578766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:31.596681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:31.781908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4d2e379bb3894a3db93feb28c95a1367'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:31.782690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4d2e379bb3894a3db93feb28c95a1367'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:31.791460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4d2e379bb3894a3db93feb28c95a1367'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:31.792096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4d2e379bb3894a3db93feb28c95a1367'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:31.893794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=5,keystone:DELETE=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:31.922167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4d2e379bb3894a3db93feb28c95a1367'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:31.960204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1036/2065] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:31 2022] DELETE /identity/v3/users/4d2e379bb3894a3db93feb28c95a1367 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:31.965128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:31.965413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:31.965872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:31.999969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:31.999969 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:32.056348 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1030/2066] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:32.067054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:32.068599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:32.126642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd025951-ca40-4f0f-9873-285985ed8102 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:32.130229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5e2015f-ad4d-487d-8d6a-a07fb04c8270 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:32.130855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2015f-ad4d-487d-8d6a-a07fb04c8270 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:32.131135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2015f-ad4d-487d-8d6a-a07fb04c8270 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:32.131404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5e2015f-ad4d-487d-8d6a-a07fb04c8270 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:32.141298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e2015f-ad4d-487d-8d6a-a07fb04c8270 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:32.142968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5e2015f-ad4d-487d-8d6a-a07fb04c8270 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:32.153162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1037/2067] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:32 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:32.624364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:32.738351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:32.873046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:32.874505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:32.887825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-458ba4a7-4513-42d5-91cd-255594070a58 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:32.892807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:32.893537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:32.894154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:32.894667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] PATH_INFO: `/v3/projects/817923d7c2464263969257ed51cc7607` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:32.898003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] RBAC: Authorizing `identity:delete_project(project_id=817923d7c2464263969257ed51cc7607)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:32.898003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:33.030223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '817923d7c2464263969257ed51cc7607', 'request_id': 'req-3551201a-0dcf-4f92-9915-61b3423a7b1e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:33.035706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '817923d7c2464263969257ed51cc7607', 'request_id': 'req-3551201a-0dcf-4f92-9915-61b3423a7b1e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:33.264999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '817923d7c2464263969257ed51cc7607', 'request_id': 'req-3551201a-0dcf-4f92-9915-61b3423a7b1e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:33.277160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] The token cache is being invalidate because project 817923d7c2464263969257ed51cc7607 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:33.277160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:33.278059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3551201a-0dcf-4f92-9915-61b3423a7b1e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:33.279604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1031/2068] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:32 2022] DELETE /identity/v3/projects/817923d7c2464263969257ed51cc7607 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:33.284457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898762bf-f293-4158-9e70-940ffd670879 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.284735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898762bf-f293-4158-9e70-940ffd670879 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.285002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-898762bf-f293-4158-9e70-940ffd670879 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.305520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-898762bf-f293-4158-9e70-940ffd670879 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:33.306754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-898762bf-f293-4158-9e70-940ffd670879 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:33.369015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1038/2069] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:33.377343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.377624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.377916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.399065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:33.399065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:33.408093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1032/2070] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:33.416202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898762bf-f293-4158-9e70-940ffd670879 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:33.426566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-898762bf-f293-4158-9e70-940ffd670879 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:33.433706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:33.434326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.434603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.434910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.435606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:33.435909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:33.437893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:33.451150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1039/2071] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:33.455309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:33.465210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a61a6f5-d546-455e-94b7-f7bb9fb8b39a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:33.470241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:33.470865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.473960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.473960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.473960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:33.473960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:33.587568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:33.596298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1033/2072] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:33 2022] POST /identity/v3/projects => generated 373 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:33.600210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:33.622495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7860134a-f443-4cb2-9ad7-d4179ede9690 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:33.697687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:33.698290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.698563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.698860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.699507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:33.701449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:33.838260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1040/2073] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:33 2022] POST /identity/v3/users => generated 287 bytes in 239 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:33.843263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:33.856870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a981ca-88eb-47d8-acf5-eab7bc396c34 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:33.868674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:33.869492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.869945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.870370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.872007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:33.873777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:33.885865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1034/2074] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:33 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:33.890842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:33.903555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-231fc3b3-22a4-476c-b6dd-d315926debc6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:33.908536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4f3961-a8a7-4509-8ba8-b566f5c0679d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:33.909193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f3961-a8a7-4509-8ba8-b566f5c0679d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.909820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f3961-a8a7-4509-8ba8-b566f5c0679d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.910122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4f3961-a8a7-4509-8ba8-b566f5c0679d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.910745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f3961-a8a7-4509-8ba8-b566f5c0679d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:33.912782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4f3961-a8a7-4509-8ba8-b566f5c0679d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:33.925547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1041/2075] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:33 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:33.929597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:33.940620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e69add1-a4ab-44ba-9e7e-9f631b5f6cf6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:33.947886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:33.948561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:33.949512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:33.950854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] PATH_INFO: `/v3/projects/e35e92f72c084d2e9f8635160b457f05/users/a2d45808777d4255b58bdcda872515c0/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:33.972416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e35e92f72c084d2e9f8635160b457f05, user_id=a2d45808777d4255b58bdcda872515c0, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:33.976061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.014031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=86,keystone:INSERT=4,keystone:DELETE=10 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:34.052147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1035/2076] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:33 2022] PUT /identity/v3/projects/e35e92f72c084d2e9f8635160b457f05/users/a2d45808777d4255b58bdcda872515c0/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:34.062335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.062621 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.062938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.071218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:34.095475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] MFA Rules not processed for user `a2d45808777d4255b58bdcda872515c0`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:34.096689 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:34.165954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1042/2077] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:34 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:34.171289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.209526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.216371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:34.271967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cc9251b-9ddb-461d-a128-a4f24d15e874 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.277978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.278605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.278947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.279221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.279857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.281792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.305227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-857c7c7f-9086-4e7d-8984-326a30bcc272 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.310510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.311386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.311664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.311961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] PATH_INFO: `/v3/users/653e9deedcac44438d7a7535e7c433d6` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.313291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] RBAC: Authorizing `identity:delete_user(user_id=653e9deedcac44438d7a7535e7c433d6)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.315143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.408995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1036/2078] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:34 2022] POST /identity/v3/projects => generated 375 bytes in 235 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:34.412960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.421857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:34.492691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9243e51-64d6-4b36-8695-372d05bb3387 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.497895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.498581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.498998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.499336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.500153 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.502344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.509681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:34.553836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '653e9deedcac44438d7a7535e7c433d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:34.554555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '653e9deedcac44438d7a7535e7c433d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:34.562282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '653e9deedcac44438d7a7535e7c433d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:34.562981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '653e9deedcac44438d7a7535e7c433d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:34.629974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1037/2079] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:34 2022] POST /identity/v3/users => generated 294 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:34.633922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.635487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:34.647738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d74cf0b-938d-43bb-8f75-69ef3c7dff42 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.653018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.653642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.653920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.654189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.654851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.656438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.659704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f746636d-b997-4f18-8273-e8c1478b1944 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '653e9deedcac44438d7a7535e7c433d6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:34.667159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1038/2080] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:34 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:34.671530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.681885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1043/2081] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:34 2022] DELETE /identity/v3/users/653e9deedcac44438d7a7535e7c433d6 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:34.682913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.683297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.683586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.685377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcf59340-26be-475e-9986-f97e30de66ee admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.705642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:34.715489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:34.770736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.771452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.772798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.772798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] PATH_INFO: `/v3/projects/a985563bd4524156850052bb619381df/users/a265ee944950427e83bdbdf4c747086b/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.792758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a985563bd4524156850052bb619381df, user_id=a265ee944950427e83bdbdf4c747086b, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.796064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1044/2082] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:34.796534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.806313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.808505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:34.871312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3dffa73-b3b7-4180-a3e3-e4a6648fff70 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.876699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.877316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.877596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.877873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.891591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.893215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:34.893771 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1039/2083] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:34 2022] PUT /identity/v3/projects/a985563bd4524156850052bb619381df/users/a265ee944950427e83bdbdf4c747086b/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 223 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:34.901645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:34.904793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1045/2084] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:34 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:34.973386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:34.985739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7522370c-37da-45f6-ae75-aab17ceece58 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:34.991946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-485a691f-7658-4715-aebc-d94f3a1e2497 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:34.992594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a691f-7658-4715-aebc-d94f3a1e2497 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:34.992876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a691f-7658-4715-aebc-d94f3a1e2497 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:34.993152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-485a691f-7658-4715-aebc-d94f3a1e2497 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:34.993852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a691f-7658-4715-aebc-d94f3a1e2497 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:34.996307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-485a691f-7658-4715-aebc-d94f3a1e2497 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:35.008565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1040/2085] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:34 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:35.014022 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:35.023831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a554a93-9f03-4560-a639-08fd081ee9aa service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:35.028673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:35.029260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:35.029587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:35.030168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] PATH_INFO: `/v3/projects/a985563bd4524156850052bb619381df/users/a265ee944950427e83bdbdf4c747086b/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:35.032135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a985563bd4524156850052bb619381df, user_id=a265ee944950427e83bdbdf4c747086b, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:35.034823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:35.046016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1046/2086] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:35 2022] PUT /identity/v3/projects/a985563bd4524156850052bb619381df/users/a265ee944950427e83bdbdf4c747086b/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:35.057009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:35.057517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:35.057822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:35.089029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] MFA Rules not processed for user `a265ee944950427e83bdbdf4c747086b`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:35.090818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:35.178014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1041/2087] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:35 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:35.191358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:35.193311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:35.269387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb7778a0-2ce4-4243-b374-087264236d42 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:35.274544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:35.275286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:35.275524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:35.278104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:35.278104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:35.358707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:35.361344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:35.362941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:35.373997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1047/2088] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:35 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:35.385108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:35.387561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:35.429341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:35.538219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e05ab4-ffe5-432f-b094-158b422f2ed8 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:35.547136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:35.548256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:35.548860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:35.550036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] PATH_INFO: `/v3/users/1bb251c2304d477dbcc13296c34edb3c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:35.552176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:35.553231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1bb251c2304d477dbcc13296c34edb3c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:35.555558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:35.659922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:35.740425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:35.741847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:35.766417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0b61282-0318-48d1-990d-32047cc445a0 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:35.774987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:35.776094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:35.777760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:35.779030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] PATH_INFO: `/v3/projects/6abe722eb0a34e98ac59acac0b02c903` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:35.780464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6abe722eb0a34e98ac59acac0b02c903)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:35.784924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:35.796109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1bb251c2304d477dbcc13296c34edb3c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:35.797552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1bb251c2304d477dbcc13296c34edb3c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:35.798360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1bb251c2304d477dbcc13296c34edb3c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:35.799600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1bb251c2304d477dbcc13296c34edb3c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:35.939919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-6393fd6c-72f0-415b-9f3f-dc240a8c2dd0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1bb251c2304d477dbcc13296c34edb3c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:35.959514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1042/2089] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:35 2022] DELETE /identity/v3/users/1bb251c2304d477dbcc13296c34edb3c => generated 0 bytes in 575 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:35.964270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:35.966475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:35.966999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:35.988698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:35.990224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:35.993510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6abe722eb0a34e98ac59acac0b02c903', 'request_id': 'req-14f4a285-6733-4cc7-8785-465fb086bd47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:36.057353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1043/2090] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:36.070433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.072972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:36.103799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6abe722eb0a34e98ac59acac0b02c903', 'request_id': 'req-14f4a285-6733-4cc7-8785-465fb086bd47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:36.116236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6abe722eb0a34e98ac59acac0b02c903', 'request_id': 'req-14f4a285-6733-4cc7-8785-465fb086bd47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:36.128515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] The token cache is being invalidate because project 6abe722eb0a34e98ac59acac0b02c903 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:36.128810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:36.129483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-14f4a285-6733-4cc7-8785-465fb086bd47 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:36.134725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1048/2091] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:35 2022] DELETE /identity/v3/projects/6abe722eb0a34e98ac59acac0b02c903 => generated 0 bytes in 393 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:36.137101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70a5ce-c174-4943-9d0c-7b08720b3706 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.137546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70a5ce-c174-4943-9d0c-7b08720b3706 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.137695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e70a5ce-c174-4943-9d0c-7b08720b3706 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.157851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23833fc6-3d8c-4ff4-8749-5a3edc9496fd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.160490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4e70a5ce-c174-4943-9d0c-7b08720b3706 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:36.162207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4e70a5ce-c174-4943-9d0c-7b08720b3706 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:36.162377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.163097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.163489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.163832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.166212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:36.179487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1049/2092] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:36.180910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.181194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.181463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.187613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.189171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.210744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:36.211924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:36.216751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1044/2093] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:36 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:36.228906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1050/2094] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:36 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:36.234373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.251187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3d718f6-b1c8-4493-a002-ad2b30113ee0 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.259194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.259883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.260901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.261325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.262815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:36.263285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.265011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.286518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1045/2095] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:37:36.293153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.308865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc5e424b-ebbf-479f-b27c-5acf0b77735e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.317581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.318398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.318741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.328766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.328766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.328766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.447352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1051/2096] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:36 2022] POST /identity/v3/projects => generated 385 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:36.448679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.459761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1c1368d-2fb8-454f-8a34-887c72701bab admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.552135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:36.575879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.577081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.577367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.577933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.578637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.580746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.705803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1046/2097] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:36 2022] POST /identity/v3/users => generated 293 bytes in 258 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:36.706150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.717096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32e1e50-063a-4efc-aaba-a0f82eac9e8e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.725284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.726187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.731810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.731810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.731810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.732734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.743697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1052/2098] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:36 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:36.748438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.759371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af90afb1-f333-47b5-9c0a-673ea4220c39 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.765802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.766485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.766953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.767362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.768092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.769756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.780699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1047/2099] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:36 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:36.784977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.796541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d519777b-2fbd-4a18-bd3b-6aefabfa55dd admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.802713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63b209a6-9a64-4976-bd44-5be095d99068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.803554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b209a6-9a64-4976-bd44-5be095d99068 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.803694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b209a6-9a64-4976-bd44-5be095d99068 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.803962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63b209a6-9a64-4976-bd44-5be095d99068 admin admin] PATH_INFO: `/v3/projects/8952a5f6136c4505b0c274048d9dd9cc/users/9cc4d0f8281a46629fa15b93c20e59ac/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.822006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b209a6-9a64-4976-bd44-5be095d99068 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8952a5f6136c4505b0c274048d9dd9cc, user_id=9cc4d0f8281a46629fa15b93c20e59ac, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.825079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63b209a6-9a64-4976-bd44-5be095d99068 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.852640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:36.854431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:36.867764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e387f3-8876-404f-90be-0c73564403b7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:36.873197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:36.875072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:36.875723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.877736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.877736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] PATH_INFO: `/v3/projects/ee72d024fa844e0fb199018e5a52887e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.877736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee72d024fa844e0fb199018e5a52887e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:36.879419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:36.940516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1053/2100] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:36 2022] PUT /identity/v3/projects/8952a5f6136c4505b0c274048d9dd9cc/users/9cc4d0f8281a46629fa15b93c20e59ac/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:36.952088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:36.952366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:36.952634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:36.979647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] MFA Rules not processed for user `9cc4d0f8281a46629fa15b93c20e59ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:36.981513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:37.050759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1054/2101] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:36 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:37.063307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:37.064985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:37.065739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee72d024fa844e0fb199018e5a52887e', 'request_id': 'req-465edec2-9d51-422c-bd5d-c054279d4e5f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.073008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee72d024fa844e0fb199018e5a52887e', 'request_id': 'req-465edec2-9d51-422c-bd5d-c054279d4e5f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.131297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568768f-0a4c-425e-8330-545a0ee7c4bd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:37.184736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:37.185538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:37.185816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:37.186082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:37.253418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:37.254794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:37.258026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee72d024fa844e0fb199018e5a52887e', 'request_id': 'req-465edec2-9d51-422c-bd5d-c054279d4e5f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.265506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1055/2102] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:37 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 203 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:37.271670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] The token cache is being invalidate because project ee72d024fa844e0fb199018e5a52887e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:37.271966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.272650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.274091 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1048/2103] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:36 2022] DELETE /identity/v3/projects/ee72d024fa844e0fb199018e5a52887e => generated 0 bytes in 422 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:37.279408 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:37.343439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21933514-d45f-421c-bfcd-b75e9ae052ad service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:37.349230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:37.349855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:37.350129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:37.350399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] PATH_INFO: `/v3/users/4085f888392d46188b32e0c7ba2b2c34` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:37.363211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] RBAC: Authorizing `identity:delete_user(user_id=4085f888392d46188b32e0c7ba2b2c34)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:37.363211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:37.437153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:37.628201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4085f888392d46188b32e0c7ba2b2c34'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.628938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4085f888392d46188b32e0c7ba2b2c34'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.643424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4085f888392d46188b32e0c7ba2b2c34'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.643424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4085f888392d46188b32e0c7ba2b2c34'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.749226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a430333e-e152-4372-a05e-200cb610affb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4085f888392d46188b32e0c7ba2b2c34'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:37.773783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1056/2104] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:37 2022] DELETE /identity/v3/users/4085f888392d46188b32e0c7ba2b2c34 => generated 0 bytes in 495 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:38.088445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:38.335653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:38.337206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:38.356975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-465edec2-9d51-422c-bd5d-c054279d4e5f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:38.422229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab625b0-b755-42fb-8cf0-4b549989c94e None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.422951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab625b0-b755-42fb-8cf0-4b549989c94e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.422951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ab625b0-b755-42fb-8cf0-4b549989c94e None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.424451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1057/2105] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:37:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:37:38.424901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:38.425654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.426008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.426363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] PATH_INFO: `/v3/projects/d576c830819f484fb2c13bf21ce92017` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.427582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d576c830819f484fb2c13bf21ce92017)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:38.429490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:38.454203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd19677e-e3d8-45d4-95b6-8b9acc07001d None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.454572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd19677e-e3d8-45d4-95b6-8b9acc07001d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.455123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd19677e-e3d8-45d4-95b6-8b9acc07001d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.477150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fd19677e-e3d8-45d4-95b6-8b9acc07001d None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:38.478344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fd19677e-e3d8-45d4-95b6-8b9acc07001d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:38.493084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:38.545663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1058/2106] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:37:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:38.552302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd576c830819f484fb2c13bf21ce92017', 'request_id': 'req-edb69949-e75b-4856-87c9-c352c832af7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.557116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee71566-99de-43da-a459-edc7ddf2889d None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.557522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee71566-99de-43da-a459-edc7ddf2889d None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.557806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eee71566-99de-43da-a459-edc7ddf2889d None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.557915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd576c830819f484fb2c13bf21ce92017', 'request_id': 'req-edb69949-e75b-4856-87c9-c352c832af7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.559222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1059/2107] 10.222.0.64 () {54 vars in 961 bytes} [Tue Sep 27 16:37:38 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Sep 27 16:37:38.564260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.564488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.564762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.585639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:38.586899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:38.646081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd576c830819f484fb2c13bf21ce92017', 'request_id': 'req-edb69949-e75b-4856-87c9-c352c832af7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.655455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1060/2108] 10.222.0.64 () {58 vars in 1058 bytes} [Tue Sep 27 16:37:38 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:38.656474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:38.657771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:38.658615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] The token cache is being invalidate because project d576c830819f484fb2c13bf21ce92017 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:38.658988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.659704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.661532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1049/2109] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:38 2022] DELETE /identity/v3/projects/d576c830819f484fb2c13bf21ce92017 => generated 0 bytes in 327 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:38.665353 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:38.666678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:38.726497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cda0d798-1f2f-4f8f-9d4d-7ebb4e33a8a0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:38.736614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb69949-e75b-4856-87c9-c352c832af7c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:38.736782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:38.737415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.737826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.738105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] PATH_INFO: `/v3/users/02e263a6105642c1815e1c59429278f7` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.739610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] RBAC: Authorizing `identity:delete_user(user_id=02e263a6105642c1815e1c59429278f7)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:38.741466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:38.741824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:38.742094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:38.742373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:38.742647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] PATH_INFO: `/v3/users/9d9458049a384442806df3243f9a50fe` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:38.743815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9d9458049a384442806df3243f9a50fe)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:38.746279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:38.857902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:38.942627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '02e263a6105642c1815e1c59429278f7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.943190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9d9458049a384442806df3243f9a50fe'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.943423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '02e263a6105642c1815e1c59429278f7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.951170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9d9458049a384442806df3243f9a50fe'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.951916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '02e263a6105642c1815e1c59429278f7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.952031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9d9458049a384442806df3243f9a50fe'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.952558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9d9458049a384442806df3243f9a50fe'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:38.952670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '02e263a6105642c1815e1c59429278f7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.023174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9d9458049a384442806df3243f9a50fe'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.037430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:39.044523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1050/2110] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:38 2022] DELETE /identity/v3/users/9d9458049a384442806df3243f9a50fe => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:39.047707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.047707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.103404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '02e263a6105642c1815e1c59429278f7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.128750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1061/2111] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:38 2022] DELETE /identity/v3/users/02e263a6105642c1815e1c59429278f7 => generated 0 bytes in 473 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:39.144096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc2f8b3-20f1-4d89-a03e-1693228bbf24 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.206898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:39.207513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.207792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.208063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.209817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.220873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:39.222462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:39.233395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1051/2112] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:39 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:39.447430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.449496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.518985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bcd6e7c-4c23-4983-8360-b10f5cdbf304 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.523629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:39.524259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.524537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.524807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] PATH_INFO: `/v3/projects/996f7fb4be7b42ceb08488b812087543` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.525903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] RBAC: Authorizing `identity:delete_project(project_id=996f7fb4be7b42ceb08488b812087543)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:39.528157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:39.548080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.549940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.567873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c9f401a-5915-45df-bb45-077898214376 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.572681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:39.573310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.573740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.574706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] PATH_INFO: `/v3/projects/dbc7e83dd3d3429aaa953500ef3ae337` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.575905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] RBAC: Authorizing `identity:delete_project(project_id=dbc7e83dd3d3429aaa953500ef3ae337)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:39.577805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:39.631415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '996f7fb4be7b42ceb08488b812087543', 'request_id': 'req-0600eb09-d305-4309-8a7d-73d63734c252'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.716630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '996f7fb4be7b42ceb08488b812087543', 'request_id': 'req-0600eb09-d305-4309-8a7d-73d63734c252'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.720305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:39.723704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '996f7fb4be7b42ceb08488b812087543', 'request_id': 'req-0600eb09-d305-4309-8a7d-73d63734c252'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.725571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dbc7e83dd3d3429aaa953500ef3ae337', 'request_id': 'req-df2c0e3f-a38d-429f-945d-657a21f2f7be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.732744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dbc7e83dd3d3429aaa953500ef3ae337', 'request_id': 'req-df2c0e3f-a38d-429f-945d-657a21f2f7be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.736028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] The token cache is being invalidate because project 996f7fb4be7b42ceb08488b812087543 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:39.736352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.737103 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0600eb09-d305-4309-8a7d-73d63734c252 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.738975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1062/2113] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:39 2022] DELETE /identity/v3/projects/996f7fb4be7b42ceb08488b812087543 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:39.744864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe7d52e-7a1d-4170-ad78-8438b4dd97a0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.745144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe7d52e-7a1d-4170-ad78-8438b4dd97a0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.745416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fe7d52e-7a1d-4170-ad78-8438b4dd97a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.766635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1fe7d52e-7a1d-4170-ad78-8438b4dd97a0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:39.767965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1fe7d52e-7a1d-4170-ad78-8438b4dd97a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.829468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1063/2114] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:39.836982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.837272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.837549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.856723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dbc7e83dd3d3429aaa953500ef3ae337', 'request_id': 'req-df2c0e3f-a38d-429f-945d-657a21f2f7be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.858213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:39.859463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.868302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1064/2115] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:39.869784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] The token cache is being invalidate because project dbc7e83dd3d3429aaa953500ef3ae337 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:39.870097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.871391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-df2c0e3f-a38d-429f-945d-657a21f2f7be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:39.872786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.873096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1052/2116] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:39 2022] DELETE /identity/v3/projects/dbc7e83dd3d3429aaa953500ef3ae337 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:39.877552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d969c01-cabd-4bdd-897c-4ebd9670ff7e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.877839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d969c01-cabd-4bdd-897c-4ebd9670ff7e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.878133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d969c01-cabd-4bdd-897c-4ebd9670ff7e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.882361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49b01bd3-7d7e-4a51-99ac-20c8ec7a5c6a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.887150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:39.887747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.888028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.888347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.889140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:39.889449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:39.891091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:39.897398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7d969c01-cabd-4bdd-897c-4ebd9670ff7e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:39.898680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7d969c01-cabd-4bdd-897c-4ebd9670ff7e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.906342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1065/2117] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:37:39.907225 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.907733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1053/2118] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:39.908825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.915536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.915820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.917127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.922380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c659da6-60bf-4216-9bac-2bcb3120cb2f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.927085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:39.927670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.927944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.928211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.928912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:39.930858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:39.936961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:39.938373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:39.947145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1054/2119] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:39.951269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:39.951482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.960520 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1580469-9eb8-4ede-bb68-5f99000cfbb3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.965319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:39.965901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:39.966175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:39.966443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:39.967102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:39.967407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:39.968984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:39.980761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1055/2120] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:37:39.985024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:39.994299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a670d65-6610-4eb3-9b63-b7bbc98021a6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:39.999742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.000305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.000581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.000851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.001455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.003356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.076144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1066/2121] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:39 2022] POST /identity/v3/projects => generated 389 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:40.082013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.119225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1056/2122] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:39 2022] POST /identity/v3/projects => generated 391 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:40.133663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.157409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e179bf2-f8d8-42b8-ab41-193864a38020 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.204252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6634dafb-be0e-4a44-9b4f-1e25f67d146f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.208904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.209509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.209781 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.210051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.210717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.212768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.218498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.219132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.219404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.219670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.220291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.222288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.325849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1057/2123] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:40 2022] POST /identity/v3/users => generated 296 bytes in 193 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:40.329802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.338274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1067/2124] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:40 2022] POST /identity/v3/users => generated 295 bytes in 257 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:40.340626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4f7f008-166e-4909-bf4e-e38dec19e9e8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.345204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.345804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.346079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.346344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.347057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.348637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.348903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.362603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1058/2125] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:40.363637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.365012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-949fb936-e338-4c48-b7d5-0592cc336b65 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.369877 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.370566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.370911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.371210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.371845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.374744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.374878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07eef7dd-dc06-4b43-a81e-96b8d1faa1ac admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.378246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.378855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.379135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.379409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.380034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.381612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.384211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1068/2126] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:40 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:40.388615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.392861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1059/2127] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:40 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:40.397317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.398312 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-122f5895-2103-4e69-bfdc-6150002cefea admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.405246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.405857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.406149 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.406406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.407099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.408738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.410362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8e85d25-8492-42ab-bcb0-04ac45b173ba admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.416184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fdbacca-1466-4d35-b184-0d26b0d16c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.416891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdbacca-1466-4d35-b184-0d26b0d16c7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.417229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdbacca-1466-4d35-b184-0d26b0d16c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.418947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fdbacca-1466-4d35-b184-0d26b0d16c7f admin admin] PATH_INFO: `/v3/projects/a8ff4b44d04942c4b2599eba4b0dafa6/users/9031384442d04ecf84b98138bb10f359/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.421740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1069/2128] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:40 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:40.426657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.436342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0180f739-8b4a-4c06-a87e-090d9ec132da admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.436495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdbacca-1466-4d35-b184-0d26b0d16c7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8ff4b44d04942c4b2599eba4b0dafa6, user_id=9031384442d04ecf84b98138bb10f359, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.439304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fdbacca-1466-4d35-b184-0d26b0d16c7f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.441419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7004a37c-4fef-4724-a755-c2ac6599be10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.442063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004a37c-4fef-4724-a755-c2ac6599be10 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.442335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004a37c-4fef-4724-a755-c2ac6599be10 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.442602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7004a37c-4fef-4724-a755-c2ac6599be10 admin admin] PATH_INFO: `/v3/projects/9288b4e62bc945479636aec307a98746/users/2f9e55a3ff7c424dbd7a58f0eb92b8ce/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.458283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004a37c-4fef-4724-a755-c2ac6599be10 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9288b4e62bc945479636aec307a98746, user_id=2f9e55a3ff7c424dbd7a58f0eb92b8ce, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.460948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7004a37c-4fef-4724-a755-c2ac6599be10 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.474891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:40.545009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1060/2129] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:40 2022] PUT /identity/v3/projects/a8ff4b44d04942c4b2599eba4b0dafa6/users/9031384442d04ecf84b98138bb10f359/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 148 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:40.556205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.556490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.556764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.584806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] MFA Rules not processed for user `9031384442d04ecf84b98138bb10f359`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:40.586052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:40.597845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1070/2130] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:40 2022] PUT /identity/v3/projects/9288b4e62bc945479636aec307a98746/users/2f9e55a3ff7c424dbd7a58f0eb92b8ce/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:40.603173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.603459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.603731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.624133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:40.625351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:40.664146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1061/2131] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:40 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:40.671925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.673308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:40.702294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1071/2132] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:40 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:40.711910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:40.713608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:40.734713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed2a85b5-dc24-44eb-a9b0-3a91d4c3596f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.775209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6add3603-d62e-4e93-934c-672c4c992f3a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:40.780601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.781211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.781494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.781768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.785727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd47743-5929-4acd-ad47-5319839fe307 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:40.786590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd47743-5929-4acd-ad47-5319839fe307 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:40.786901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd47743-5929-4acd-ad47-5319839fe307 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:40.787176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd47743-5929-4acd-ad47-5319839fe307 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:40.791595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.793414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.803183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1072/2133] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:40 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:40.847632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd47743-5929-4acd-ad47-5319839fe307 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:40.849063 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd47743-5929-4acd-ad47-5319839fe307 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:40.859221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1062/2134] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:40 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 188 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:41.264145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=4,keystone:DELETE=8 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:41.460958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:41.464323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:41.477171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a679471a-9290-469b-8970-27a167e6859d service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:41.482537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:41.483194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:41.483543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:41.483823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] PATH_INFO: `/v3/users/a2d45808777d4255b58bdcda872515c0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:41.485083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a2d45808777d4255b58bdcda872515c0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:41.487017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:41.746238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a2d45808777d4255b58bdcda872515c0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:41.746947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a2d45808777d4255b58bdcda872515c0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:41.755323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a2d45808777d4255b58bdcda872515c0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:41.756119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a2d45808777d4255b58bdcda872515c0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:41.846194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a2d45808777d4255b58bdcda872515c0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:41.866992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1073/2135] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:41 2022] DELETE /identity/v3/users/a2d45808777d4255b58bdcda872515c0 => generated 0 bytes in 407 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:41.871628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:41.872440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:41.872716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:41.895343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:41.896505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:41.947488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:INSERT=4,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:41.963787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1063/2136] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:41.974300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:41.975811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:42.049135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304a0264-0031-4755-8eef-b29e9927fa44 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:42.050065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:42.050799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:42.050971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:42.051288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:42.062349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:42.063857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:42.073498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1074/2137] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:41 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:42.612973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:42.614559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:42.630475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dd15f82-9b4f-4bda-9e26-0bc99eb4567d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:42.637528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:42.638214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:42.638558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:42.645714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] PATH_INFO: `/v3/projects/e35e92f72c084d2e9f8635160b457f05` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:42.645714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e35e92f72c084d2e9f8635160b457f05)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:42.645714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:42.852840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=2,keystone:SELECT=97,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:42.876657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e35e92f72c084d2e9f8635160b457f05', 'request_id': 'req-4094cd96-72b3-4381-bb8f-0d245668a9c0'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:42.883011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e35e92f72c084d2e9f8635160b457f05', 'request_id': 'req-4094cd96-72b3-4381-bb8f-0d245668a9c0'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:42.994358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e35e92f72c084d2e9f8635160b457f05', 'request_id': 'req-4094cd96-72b3-4381-bb8f-0d245668a9c0'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.017862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] The token cache is being invalidate because project e35e92f72c084d2e9f8635160b457f05 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:43.017862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.017862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.017862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1064/2138] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:42 2022] DELETE /identity/v3/projects/e35e92f72c084d2e9f8635160b457f05 => generated 0 bytes in 398 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:43.018224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:43.025101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535f0575-c194-4c90-9652-22ac38c86199 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:43.099000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:43.099646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:43.099921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:43.100195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] PATH_INFO: `/v3/users/a265ee944950427e83bdbdf4c747086b` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:43.109497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a265ee944950427e83bdbdf4c747086b)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:43.109497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:43.321044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a265ee944950427e83bdbdf4c747086b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.321706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a265ee944950427e83bdbdf4c747086b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.331034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a265ee944950427e83bdbdf4c747086b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.332002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a265ee944950427e83bdbdf4c747086b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.425504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ffd65cdf-ccc4-4d43-837e-499c15102811 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a265ee944950427e83bdbdf4c747086b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:43.465064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1075/2139] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:43 2022] DELETE /identity/v3/users/a265ee944950427e83bdbdf4c747086b => generated 0 bytes in 451 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:44.047542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.049695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:44.063756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4094cd96-72b3-4381-bb8f-0d245668a9c0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:44.107105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:44.170652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:44.177026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.177026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.177026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] PATH_INFO: `/v3/projects/a985563bd4524156850052bb619381df` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.177026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] RBAC: Authorizing `identity:delete_project(project_id=a985563bd4524156850052bb619381df)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:44.177026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:44.281033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a985563bd4524156850052bb619381df', 'request_id': 'req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:44.287671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a985563bd4524156850052bb619381df', 'request_id': 'req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:44.366535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a985563bd4524156850052bb619381df', 'request_id': 'req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:44.380857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] The token cache is being invalidate because project a985563bd4524156850052bb619381df was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:44.381466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:44.382219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-abaddfe4-e897-4725-9b3a-c7e7ddfbeadd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:44.389445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1065/2140] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:44 2022] DELETE /identity/v3/projects/a985563bd4524156850052bb619381df => generated 0 bytes in 339 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:44.392694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ae684-031f-4f36-b181-bf1a772843df None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.393691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ae684-031f-4f36-b181-bf1a772843df None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.394580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-015ae684-031f-4f36-b181-bf1a772843df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.417582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-015ae684-031f-4f36-b181-bf1a772843df None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:44.419495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-015ae684-031f-4f36-b181-bf1a772843df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:44.492828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1076/2141] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:44.501625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.501868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.502344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.522700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:44.523936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:44.534810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1066/2142] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:44.537513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ae684-031f-4f36-b181-bf1a772843df None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.547758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-015ae684-031f-4f36-b181-bf1a772843df None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:44.552762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:44.553429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.553791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.554063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.554718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:44.555856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:44.556760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:44.569132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1077/2143] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:44.574004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.583656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40f10136-1524-492b-b50f-fc3e448e63f0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:44.588425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:44.589840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.590206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.590413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.591068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:44.592973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:44.710290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1067/2144] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:44 2022] POST /identity/v3/projects => generated 385 bytes in 128 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:44.710459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.715678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47d7e775-3d2e-46e9-9330-68af3b9fe23b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:44.777026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:44.777823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.778099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.778388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.779050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:44.780986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:44.872656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1078/2145] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:44 2022] POST /identity/v3/users => generated 293 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:44.874114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.890408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bade5c5c-abdd-49d6-b58e-1c3e0e7cdd44 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:44.895666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:44.896288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.896617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.897062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.897845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:44.899847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:44.916362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.917009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1068/2146] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:44.928464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a94951-2c2a-4dcb-9cc1-a2e192b9edc4 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:44.948050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3ad799c-1546-4a6d-9c88-599be71f25c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:44.950043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ad799c-1546-4a6d-9c88-599be71f25c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:44.952241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ad799c-1546-4a6d-9c88-599be71f25c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:44.952741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3ad799c-1546-4a6d-9c88-599be71f25c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:44.953601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ad799c-1546-4a6d-9c88-599be71f25c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:44.956428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3ad799c-1546-4a6d-9c88-599be71f25c7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:44.982063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1079/2147] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:44 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:44.982967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:44.997139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab46304-f8f0-409c-8aa8-c356b6ebca0a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:45.013642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:45.014421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:45.014945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:45.015229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] PATH_INFO: `/v3/projects/8d9efb801278419ea0e634b764c4011c/users/931a313fe51e4ed89f5cdfd1210c1f1c/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:45.032327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8d9efb801278419ea0e634b764c4011c, user_id=931a313fe51e4ed89f5cdfd1210c1f1c, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:45.035437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:45.059984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:45.115477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1069/2148] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:44 2022] PUT /identity/v3/projects/8d9efb801278419ea0e634b764c4011c/users/931a313fe51e4ed89f5cdfd1210c1f1c/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 133 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:45.126066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d68c504-0404-4607-b478-0f124c83c320 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:45.126369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d68c504-0404-4607-b478-0f124c83c320 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:45.126675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d68c504-0404-4607-b478-0f124c83c320 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:45.157639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2d68c504-0404-4607-b478-0f124c83c320 None None] MFA Rules not processed for user `931a313fe51e4ed89f5cdfd1210c1f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:45.159009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2d68c504-0404-4607-b478-0f124c83c320 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:45.248579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1080/2149] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:45 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:45.261937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:45.263985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:45.337858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f08a96f0-2637-4e38-a378-42f9baf92321 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:45.342930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c091084-698e-4c08-9b53-8ed7358326ac service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:45.343963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c091084-698e-4c08-9b53-8ed7358326ac service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:45.344559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c091084-698e-4c08-9b53-8ed7358326ac service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:45.345033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c091084-698e-4c08-9b53-8ed7358326ac service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:45.355388 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c091084-698e-4c08-9b53-8ed7358326ac service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:45.356852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c091084-698e-4c08-9b53-8ed7358326ac service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:45.367355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1070/2150] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:45 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:46.168426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:47.079437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:47.079998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:47.080094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:47.090971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:47.114562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] MFA Rules not processed for user `2f9e55a3ff7c424dbd7a58f0eb92b8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:47.115757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:47.201460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1081/2151] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:47 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:47.212715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09154b0-d7a1-4f13-bfbe-43719abe22a3 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:47.213190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09154b0-d7a1-4f13-bfbe-43719abe22a3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:47.213280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f09154b0-d7a1-4f13-bfbe-43719abe22a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:47.235373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f09154b0-d7a1-4f13-bfbe-43719abe22a3 None None] MFA Rules not processed for user `2f9e55a3ff7c424dbd7a58f0eb92b8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:47.236597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f09154b0-d7a1-4f13-bfbe-43719abe22a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:47.243953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1071/2152] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:47 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:47.264024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:47.265468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:47.351782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9baa4c3-43a8-4525-854a-f9909c335fb9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:47.361503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55364b07-ec82-4533-a059-aa9ac0e3f2d1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:47.361503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55364b07-ec82-4533-a059-aa9ac0e3f2d1 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:47.361503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55364b07-ec82-4533-a059-aa9ac0e3f2d1 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:47.361503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55364b07-ec82-4533-a059-aa9ac0e3f2d1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:47.368586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55364b07-ec82-4533-a059-aa9ac0e3f2d1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:47.370124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55364b07-ec82-4533-a059-aa9ac0e3f2d1 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:47.379026 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1082/2153] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:47 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:47.408774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:47.597249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:47.678716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dd628-d0f1-4b62-9431-b5e4070d7ab3 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:47.679068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dd628-d0f1-4b62-9431-b5e4070d7ab3 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:47.679345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-406dd628-d0f1-4b62-9431-b5e4070d7ab3 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:47.680685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1072/2154] 10.222.0.64 () {54 vars in 912 bytes} [Tue Sep 27 16:37:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:37:47.686457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbda38c-18fc-4be0-bdeb-5c827749da3a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:47.686833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbda38c-18fc-4be0-bdeb-5c827749da3a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:47.687106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fbda38c-18fc-4be0-bdeb-5c827749da3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:47.708480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0fbda38c-18fc-4be0-bdeb-5c827749da3a None None] MFA Rules not processed for user `28f0f9fef2f54e56a7a92d077c25d2b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:47.709451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0fbda38c-18fc-4be0-bdeb-5c827749da3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:47.766542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1083/2155] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:37:47 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:47.778585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dd628-d0f1-4b62-9431-b5e4070d7ab3 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:47.780341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-406dd628-d0f1-4b62-9431-b5e4070d7ab3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:47.793847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-406dd628-d0f1-4b62-9431-b5e4070d7ab3 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:47.798126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:47.799025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:47.799304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:47.799576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:47.809299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:47.810877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:47.820466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1073/2156] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:47 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:48.316194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:48.370221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:48.410669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8d9f4a-6522-4971-ba3e-37012d5fa000 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:48.411359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8d9f4a-6522-4971-ba3e-37012d5fa000 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:48.411708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c8d9f4a-6522-4971-ba3e-37012d5fa000 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:48.444415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1c8d9f4a-6522-4971-ba3e-37012d5fa000 None None] MFA Rules not processed for user `2f9e55a3ff7c424dbd7a58f0eb92b8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:48.444415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1c8d9f4a-6522-4971-ba3e-37012d5fa000 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:48.456695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1084/2157] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:48 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:48.703280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:49.659731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:51.481594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:51.783184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:52.080486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=87,keystone:DELETE=9,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:52.301048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:53.001947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:53.109802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159bccfc-5755-4ee8-835d-90d3a18c7ea6 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:53.115670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:53.116375 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:53.121773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:53.121773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] PATH_INFO: `/v3/users/931a313fe51e4ed89f5cdfd1210c1f1c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:53.121773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] RBAC: Authorizing `identity:delete_user(user_id=931a313fe51e4ed89f5cdfd1210c1f1c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:53.121773 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:53.357823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '931a313fe51e4ed89f5cdfd1210c1f1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:53.364308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:53.366320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '931a313fe51e4ed89f5cdfd1210c1f1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:53.367090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '931a313fe51e4ed89f5cdfd1210c1f1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:53.367754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '931a313fe51e4ed89f5cdfd1210c1f1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:53.424921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '931a313fe51e4ed89f5cdfd1210c1f1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:53.445849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1074/2158] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:53 2022] DELETE /identity/v3/users/931a313fe51e4ed89f5cdfd1210c1f1c => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:53.451199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:53.451663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:53.451790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:53.471935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:53.473144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:53.550185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1085/2159] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:53.560057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:53.561473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:53.618892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41ec762-2e6d-4714-a71e-c547d325a293 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:53.623481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b91b7b25-343e-426e-918e-b67542d6a166 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:53.624086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91b7b25-343e-426e-918e-b67542d6a166 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:53.624361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91b7b25-343e-426e-918e-b67542d6a166 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:53.624631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b91b7b25-343e-426e-918e-b67542d6a166 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:53.633690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91b7b25-343e-426e-918e-b67542d6a166 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:53.635364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b91b7b25-343e-426e-918e-b67542d6a166 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:53.645805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1075/2160] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:54.173664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:54.175116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:54.187920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e8f782-29a5-4cc2-b567-cfe212dccebd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:54.192768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:54.193665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.193944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.194218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] PATH_INFO: `/v3/projects/8d9efb801278419ea0e634b764c4011c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.195372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8d9efb801278419ea0e634b764c4011c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:54.197380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:54.359670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8d9efb801278419ea0e634b764c4011c', 'request_id': 'req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:54.447623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8d9efb801278419ea0e634b764c4011c', 'request_id': 'req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:54.454818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8d9efb801278419ea0e634b764c4011c', 'request_id': 'req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:54.466481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] The token cache is being invalidate because project 8d9efb801278419ea0e634b764c4011c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:54.466789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:54.467445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-f9f5aa3b-d09f-4054-934a-d2d4de8f0f66 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:54.469003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1086/2161] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:54 2022] DELETE /identity/v3/projects/8d9efb801278419ea0e634b764c4011c => generated 0 bytes in 296 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:54.473888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.474314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.474430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.494257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:54.495607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:54.554198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1076/2162] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:54.565068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.565068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.565068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.582325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:54.583553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:54.592048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1087/2163] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:54.596510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:54.606856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3b69812-6d6c-46bd-b28b-09c39d168272 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:54.612070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:54.612679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.612960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.613235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.613914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:54.614222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:54.615940 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:54.628423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1077/2164] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:54.632927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:54.642309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c3dad3d-cb16-4471-94dc-7454e33abdcf None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:54.646950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:54.647538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.647815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.648082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.648705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:54.650604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:54.791400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:54.798539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1088/2165] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:54 2022] POST /identity/v3/projects => generated 373 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:54.802726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:54.812936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b370a069-c977-4f25-bd3b-629ee7a56a26 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:54.871821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:54.872429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.872705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.872973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.874366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:54.875539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:54.884487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=2,keystone:DELETE=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:54.957202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1078/2166] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:54 2022] POST /identity/v3/users => generated 287 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:37:54.961423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:54.972572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d52612-dc56-4449-b0b6-1eada4b03119 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:54.977396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:54.978122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:54.978403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:54.978671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:54.979344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:54.981040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:54.991328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1089/2167] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:54 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:54.995229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:55.006048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5907574b-ecbb-4314-8a18-17b62c3af88f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:55.011339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d75242c5-d0f6-46ad-844e-f7490d42e684 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:55.012048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75242c5-d0f6-46ad-844e-f7490d42e684 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:55.012332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75242c5-d0f6-46ad-844e-f7490d42e684 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:55.012604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d75242c5-d0f6-46ad-844e-f7490d42e684 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:55.013242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75242c5-d0f6-46ad-844e-f7490d42e684 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:55.014857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d75242c5-d0f6-46ad-844e-f7490d42e684 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:55.025563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1079/2168] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:54 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:55.032121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:55.042458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-497367b9-88cd-467b-a3e0-acc2c6a5c3f1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:55.047046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:55.047712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:55.047988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:55.048258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] PATH_INFO: `/v3/projects/3e379e32c618473bbe866b18cc1e81c3/users/addc1913f3324438aa9d3a626e7fae70/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:55.062832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e379e32c618473bbe866b18cc1e81c3, user_id=addc1913f3324438aa9d3a626e7fae70, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:55.065556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:55.160238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1090/2169] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:55 2022] PUT /identity/v3/projects/3e379e32c618473bbe866b18cc1e81c3/users/addc1913f3324438aa9d3a626e7fae70/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:55.171057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:55.171339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:55.171612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:55.200333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] MFA Rules not processed for user `addc1913f3324438aa9d3a626e7fae70`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:55.201528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:55.263936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1080/2170] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:55 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:55.274480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:55.276169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:55.332319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3246dd4f-943b-4ce2-9a57-f60ac154c266 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:55.336253 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:55.336854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:55.337133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:55.337405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:55.346486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:55.347948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:55.356921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1091/2171] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:55 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:56.487420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:56.488985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:56.558129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-553253c5-b9f7-4d0b-afdf-464d87f61b5f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:56.562683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:56.563339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:56.563619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:56.563891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] PATH_INFO: `/v3/users/7a840f777cf24fda888ae3cfc5293941` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:56.565000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a840f777cf24fda888ae3cfc5293941)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:56.567043 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:56.850367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:56.852468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:56.882421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdd53885-4ed0-46bc-8ab4-978597afab67 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:56.893954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:56.903566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:56.904058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:56.904504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] PATH_INFO: `/v3/users/9cc4d0f8281a46629fa15b93c20e59ac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:56.906105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] RBAC: Authorizing `identity:delete_user(user_id=9cc4d0f8281a46629fa15b93c20e59ac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:56.908237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:56.956474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a840f777cf24fda888ae3cfc5293941'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:56.969987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a840f777cf24fda888ae3cfc5293941'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:56.971493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a840f777cf24fda888ae3cfc5293941'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:56.972409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a840f777cf24fda888ae3cfc5293941'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.168939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8c4f9d4a-0b21-45a8-84de-26b585cae1bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a840f777cf24fda888ae3cfc5293941'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.178141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=3,keystone:SELECT=88,keystone:DELETE=9 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:57.212269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1081/2172] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:56 2022] DELETE /identity/v3/users/7a840f777cf24fda888ae3cfc5293941 => generated 0 bytes in 725 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:57.225683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:57.225925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:57.226206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:57.270365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:57.271888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:57.298958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9cc4d0f8281a46629fa15b93c20e59ac'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.300072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9cc4d0f8281a46629fa15b93c20e59ac'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.316981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9cc4d0f8281a46629fa15b93c20e59ac'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.317711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9cc4d0f8281a46629fa15b93c20e59ac'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.380922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1082/2173] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:57.407523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:57.409417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:57.436849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9154e7af-f969-414f-9fcb-fbea7c147d2f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9cc4d0f8281a46629fa15b93c20e59ac'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:57.481049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1092/2174] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:56 2022] DELETE /identity/v3/users/9cc4d0f8281a46629fa15b93c20e59ac => generated 0 bytes in 631 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:57.485567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:57.485702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:57.486246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:57.515653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a905387-7871-4ccc-8ffd-d23ce2b9f2d7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:57.528056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:57.529510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:57.583897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:57.623794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1093/2175] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:57 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:57.633825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:57.640144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:57.641110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:57.663898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:57.663898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:57.664212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2b2013-ed08-46af-acbc-45340018a8fe None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:57.664212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:57.664212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:57.664212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:57.664510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:57.664579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:57.665804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:57.675020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:57.676438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:57.678838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1083/2176] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 272 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:57.686816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1094/2177] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:57 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:37:58.462649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:58.464595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:58.477569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85c9b763-2687-41d4-9d2b-e84d5e7ea35c service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:58.482535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:58.483186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.483463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.483776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] PATH_INFO: `/v3/projects/69108d4306174c54ad19214c61b21748` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.485439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=69108d4306174c54ad19214c61b21748)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:58.487629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:58.493359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:58.496500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:58.517457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b490977-444b-45b7-97e7-358ad296d160 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:58.523760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:58.524391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.524704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.525485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] PATH_INFO: `/v3/projects/8952a5f6136c4505b0c274048d9dd9cc` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.531186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] RBAC: Authorizing `identity:delete_project(project_id=8952a5f6136c4505b0c274048d9dd9cc)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:58.533625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:58.614628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:58.618316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '69108d4306174c54ad19214c61b21748', 'request_id': 'req-58e4d65a-e2d8-408f-be66-45437feab9d2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.625970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '69108d4306174c54ad19214c61b21748', 'request_id': 'req-58e4d65a-e2d8-408f-be66-45437feab9d2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.642606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8952a5f6136c4505b0c274048d9dd9cc', 'request_id': 'req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.714620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '69108d4306174c54ad19214c61b21748', 'request_id': 'req-58e4d65a-e2d8-408f-be66-45437feab9d2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.727269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] The token cache is being invalidate because project 69108d4306174c54ad19214c61b21748 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:58.727750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.732867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-58e4d65a-e2d8-408f-be66-45437feab9d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.734752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1084/2178] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:58 2022] DELETE /identity/v3/projects/69108d4306174c54ad19214c61b21748 => generated 0 bytes in 273 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:58.739862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598157b-1bf4-4244-b75f-802fb0029b68 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.740152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598157b-1bf4-4244-b75f-802fb0029b68 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.741334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9598157b-1bf4-4244-b75f-802fb0029b68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.767058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9598157b-1bf4-4244-b75f-802fb0029b68 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:58.768329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9598157b-1bf4-4244-b75f-802fb0029b68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:58.771258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8952a5f6136c4505b0c274048d9dd9cc', 'request_id': 'req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.779224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8952a5f6136c4505b0c274048d9dd9cc', 'request_id': 'req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.791200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] The token cache is being invalidate because project 8952a5f6136c4505b0c274048d9dd9cc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:37:58.791565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.793042 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cfc7549b-8c20-44cf-be7c-ed1a920c23ef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:37:58.794641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1095/2179] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:37:58 2022] DELETE /identity/v3/projects/8952a5f6136c4505b0c274048d9dd9cc => generated 0 bytes in 302 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:37:58.814028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a30558-e3f5-40c5-96de-230ff500419f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.814315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a30558-e3f5-40c5-96de-230ff500419f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.814647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6a30558-e3f5-40c5-96de-230ff500419f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.834530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c6a30558-e3f5-40c5-96de-230ff500419f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:58.835866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c6a30558-e3f5-40c5-96de-230ff500419f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:58.842759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1085/2180] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:58.843981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1096/2181] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:58.851628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.851944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.852089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.852220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.852317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.852585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.872810 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:58.873915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:58.874054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:58.875127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:58.881779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1097/2182] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:37:58.883109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1086/2183] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:58.886981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:58.887143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:58.896191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83760ce5-891d-4286-bdac-fcfefe7bbf05 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:58.899739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50a41ee-e548-4938-8b03-0d0db2302e2a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:58.901776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:58.901907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:58.902381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.902624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.902845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.902937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.903019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.903099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.903732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:58.903847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:37:58.904258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:58.904364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:58.905636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:58.905742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:58.924006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1098/2184] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:58.924289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1087/2185] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:37:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:37:58.925199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:58.926256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:58.936168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88aba01a-e407-452c-9b24-0647f63b4902 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:58.940257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce7f347-8005-4c64-908f-475a785628d6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:58.940811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:58.941401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.941688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.941943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.942593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:58.944566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:58.945959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:58.946892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:58.947220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:58.947581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:58.948376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:58.950325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.243693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1099/2186] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:58 2022] POST /identity/v3/projects => generated 373 bytes in 319 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:59.248727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.257820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1088/2187] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:37:58 2022] POST /identity/v3/projects => generated 371 bytes in 331 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:59.264639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.314887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d9048c-1ef5-40f4-aad7-f6d8a31c6cd8 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.362690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b404dbf3-ac29-4361-b8e0-ab04f27b0b33 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.381342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.381342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.381342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.381342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.381724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.381724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.381902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:59.405322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.405928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.406206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.406474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.407135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.409066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.452025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1089/2188] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:59 2022] POST /identity/v3/users => generated 286 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:59.456132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.466279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a975d56f-d88a-49db-89b8-d31f618616e5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.470950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.471541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.471819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.472090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.472723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.474304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.484843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1090/2189] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:59.488570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1100/2190] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:37:59 2022] POST /identity/v3/users => generated 287 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:37:59.496394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.496868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.506729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb6c350a-0c8f-4085-9f29-98a11cb5dd27 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.507079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99b874dc-45af-43b7-80c2-6846ffc9c17a admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.511324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.511913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.512050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.512188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.512458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.512634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.512911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.513082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.513180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.513801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.515196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.515508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.528618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1101/2191] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:37:59.528809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1091/2192] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:59.529814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.529933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.539982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6657cb55-c7e1-4b08-9884-dd0563e362cb admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.540169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba717fd6-3e4d-407a-8c7c-2dea6c852acb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.545452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.545722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b043864-cc05-4239-b954-05f348be30c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.546049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.546328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.546462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b043864-cc05-4239-b954-05f348be30c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.546602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.546735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b043864-cc05-4239-b954-05f348be30c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.546912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b043864-cc05-4239-b954-05f348be30c6 admin admin] PATH_INFO: `/v3/projects/4db73a0563ac4a24b13cbef7779ec87c/users/fba6c458d562457b8577c4e0fada0f47/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.547308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.549067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.561370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1102/2193] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:37:59 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:37:59.563805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b043864-cc05-4239-b954-05f348be30c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4db73a0563ac4a24b13cbef7779ec87c, user_id=fba6c458d562457b8577c4e0fada0f47, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.565434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.566826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b043864-cc05-4239-b954-05f348be30c6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.574831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e9a06e-44d6-4d1c-884e-baf736e3d89d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.579415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1456a82-b7e8-4c0f-afd9-15df882922d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.580042 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1456a82-b7e8-4c0f-afd9-15df882922d9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.580320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1456a82-b7e8-4c0f-afd9-15df882922d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.580592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1456a82-b7e8-4c0f-afd9-15df882922d9 admin admin] PATH_INFO: `/v3/projects/020f8f15edbc422e80de73875ff2786f/users/17b84746283d497894027f6f01e51606/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.595402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1456a82-b7e8-4c0f-afd9-15df882922d9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=020f8f15edbc422e80de73875ff2786f, user_id=17b84746283d497894027f6f01e51606, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.598083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1456a82-b7e8-4c0f-afd9-15df882922d9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.628334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1092/2194] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:59 2022] PUT /identity/v3/projects/4db73a0563ac4a24b13cbef7779ec87c/users/fba6c458d562457b8577c4e0fada0f47/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:59.631000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:37:59.638800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.639103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.639381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.668249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] MFA Rules not processed for user `fba6c458d562457b8577c4e0fada0f47`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:59.669460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:59.682712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1103/2195] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:37:59 2022] PUT /identity/v3/projects/020f8f15edbc422e80de73875ff2786f/users/17b84746283d497894027f6f01e51606/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:37:59.693260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.693595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.693923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.723309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] MFA Rules not processed for user `17b84746283d497894027f6f01e51606`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:37:59.724552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:59.736289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1093/2196] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:59 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:59.749655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.751107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:59.791393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1104/2197] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:37:59 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:37:59.802832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:37:59.804199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:37:59.811555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03490e0-a669-445b-9115-39d1f49803c9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.870508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8e3804-d4c7-47f5-876c-44e3f319a474 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:37:59.877816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.878491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.880159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.880159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.884784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2d93bab-2a5b-4edd-a11f-8f313fcac99a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:37:59.885658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d93bab-2a5b-4edd-a11f-8f313fcac99a service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:37:59.886669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d93bab-2a5b-4edd-a11f-8f313fcac99a service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:37:59.887690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2d93bab-2a5b-4edd-a11f-8f313fcac99a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:37:59.890058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.890466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.903190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1105/2198] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:59 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:59.953921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d93bab-2a5b-4edd-a11f-8f313fcac99a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:37:59.955759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2d93bab-2a5b-4edd-a11f-8f313fcac99a service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:37:59.966334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1094/2199] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:37:59 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 217 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:37:59.996696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:00.070769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3598396f-97d0-4f6e-a268-f1d0aa7412e5 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:00.076420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:00.076420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:00.076800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:00.076898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] PATH_INFO: `/v3/users/fba6c458d562457b8577c4e0fada0f47` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:00.078048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fba6c458d562457b8577c4e0fada0f47)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:00.080019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:00.333725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fba6c458d562457b8577c4e0fada0f47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:00.334304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fba6c458d562457b8577c4e0fada0f47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:00.342001 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fba6c458d562457b8577c4e0fada0f47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:00.342670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fba6c458d562457b8577c4e0fada0f47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:00.405527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fba6c458d562457b8577c4e0fada0f47'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:00.426123 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1106/2200] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:37:59 2022] DELETE /identity/v3/users/fba6c458d562457b8577c4e0fada0f47 => generated 0 bytes in 431 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:00.430335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:00.430619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:00.430930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:00.452149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:00.453372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:00.516383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1095/2201] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:00.526079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:00.528296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:00.587961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33288b11-03e2-45b1-8b7f-f17d6a00f964 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:00.590224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1bddce9-0c01-4a92-8069-20479725e7fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:00.590753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bddce9-0c01-4a92-8069-20479725e7fa service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:00.591054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bddce9-0c01-4a92-8069-20479725e7fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:00.591325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1bddce9-0c01-4a92-8069-20479725e7fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:00.600572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bddce9-0c01-4a92-8069-20479725e7fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:00.601959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1bddce9-0c01-4a92-8069-20479725e7fa service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:00.612630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1107/2202] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:01.053599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.060470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:01.075665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b23bf933-a7ef-4182-b574-46c1054ce401 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:01.081077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:01.081687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.082117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.082393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] PATH_INFO: `/v3/projects/4db73a0563ac4a24b13cbef7779ec87c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.083543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4db73a0563ac4a24b13cbef7779ec87c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:01.085521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:01.117402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:01.205271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4db73a0563ac4a24b13cbef7779ec87c', 'request_id': 'req-32350751-199d-401d-bebc-e00ede6647c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:01.211233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4db73a0563ac4a24b13cbef7779ec87c', 'request_id': 'req-32350751-199d-401d-bebc-e00ede6647c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:01.294249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4db73a0563ac4a24b13cbef7779ec87c', 'request_id': 'req-32350751-199d-401d-bebc-e00ede6647c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:01.306092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] The token cache is being invalidate because project 4db73a0563ac4a24b13cbef7779ec87c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:01.309263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:01.309263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-32350751-199d-401d-bebc-e00ede6647c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:01.309263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1096/2203] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:01 2022] DELETE /identity/v3/projects/4db73a0563ac4a24b13cbef7779ec87c => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:01.313755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.313949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.314233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.337675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:01.340835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:01.413624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1108/2204] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:01.423144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.423458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.423803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.464468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:01.465698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:01.474181 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1097/2205] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:01 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 52 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:01.478643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.492728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc4a4c56-6225-4ceb-bc48-6a0726af3ff2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:01.503324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:01.504315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.504754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.505300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.506383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:38:01.506812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:01.510319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:01.528362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1109/2206] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:38:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 50 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:38:01.531531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.546910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e7382fa-a885-4ab7-8a80-cbba3882f2ed None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:01.552300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:01.552925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.553228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.553500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.554129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:01.556155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:01.600006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:01.660159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.665646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1098/2207] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:01 2022] POST /identity/v3/projects => generated 375 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:38:01.671825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2866a859-34b9-4d52-b692-9de40fda3de3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:01.758187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:01.759085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.759434 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.759795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.761153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:01.763982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:01.909169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1110/2208] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:01 2022] POST /identity/v3/users => generated 288 bytes in 249 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:01.913358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.924032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbe25408-0db3-4e32-901c-3b2638259260 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:01.929947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:01.929947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.929947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.929947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.930407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:01.932054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:01.942383 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1099/2209] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:01.946420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.960147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-837abe94-951d-4c98-af44-2f9784964951 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:01.966839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32df8049-8882-413d-8178-c41d39c724da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:01.967555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32df8049-8882-413d-8178-c41d39c724da admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:01.967918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32df8049-8882-413d-8178-c41d39c724da admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:01.968194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32df8049-8882-413d-8178-c41d39c724da admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:01.968860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32df8049-8882-413d-8178-c41d39c724da admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:01.970515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32df8049-8882-413d-8178-c41d39c724da admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:01.981667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1111/2210] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:01 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:01.985978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:01.996536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac5ecdea-7f46-49a9-8ccd-8d33c9fcdb30 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:02.001074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:02.001724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:02.002002 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:02.002274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] PATH_INFO: `/v3/projects/808169451dd64aa1a6a656ed9abbe5a0/users/e1036377d4d94bb3a929aee17ca5eeec/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:02.022853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=808169451dd64aa1a6a656ed9abbe5a0, user_id=e1036377d4d94bb3a929aee17ca5eeec, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:02.025599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:02.112064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1100/2211] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:01 2022] PUT /identity/v3/projects/808169451dd64aa1a6a656ed9abbe5a0/users/e1036377d4d94bb3a929aee17ca5eeec/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:02.126460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c3f32-86ce-48bc-9604-fb9df16778b1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:02.126743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c3f32-86ce-48bc-9604-fb9df16778b1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:02.127753 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27c3f32-86ce-48bc-9604-fb9df16778b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:02.163687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a27c3f32-86ce-48bc-9604-fb9df16778b1 None None] MFA Rules not processed for user `e1036377d4d94bb3a929aee17ca5eeec`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:02.165128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a27c3f32-86ce-48bc-9604-fb9df16778b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:02.248494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1112/2212] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:02 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:02.387743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:02.389722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:02.426412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:02.489257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e10ce385-f48d-4c01-a25d-5caf3d1c4ea3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:02.490911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:02.491782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:02.492256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:02.492529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:02.504059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:02.511603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:02.516805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1101/2213] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:02 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:02.749441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:04.251699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:04.488955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1365e-77a4-45d4-a436-e4a1dea9455b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:04.489394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1365e-77a4-45d4-a436-e4a1dea9455b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:04.489916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68a1365e-77a4-45d4-a436-e4a1dea9455b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:04.520783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-68a1365e-77a4-45d4-a436-e4a1dea9455b None None] MFA Rules not processed for user `e1036377d4d94bb3a929aee17ca5eeec`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:04.523342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-68a1365e-77a4-45d4-a436-e4a1dea9455b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:04.536610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1113/2214] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:04 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:05.013106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:05.017131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:05.106283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d40017d4-0b66-4bc5-b203-df76f1659f0b service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:05.111244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:05.111922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:05.115364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:05.115364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] PATH_INFO: `/v3/users/17b84746283d497894027f6f01e51606` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:05.115364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] RBAC: Authorizing `identity:delete_user(user_id=17b84746283d497894027f6f01e51606)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:05.115888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:05.397676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '17b84746283d497894027f6f01e51606'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:05.406454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '17b84746283d497894027f6f01e51606'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:05.407665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '17b84746283d497894027f6f01e51606'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:05.408538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '17b84746283d497894027f6f01e51606'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:05.533376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '17b84746283d497894027f6f01e51606'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:05.566848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1102/2215] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:38:05 2022] DELETE /identity/v3/users/17b84746283d497894027f6f01e51606 => generated 0 bytes in 554 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:05.571510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:05.571510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:05.572015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:05.592077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:05.593464 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:05.674521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1114/2216] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:05.685565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:05.687256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:05.710000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:INSERT=2,keystone:SELECT=88,keystone:DELETE=10 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:05.752072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e61d6c1d-7af7-414a-8045-0544d5a8f824 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:05.777595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657a460e-230c-4b4f-b332-d12657e20737 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:05.780880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657a460e-230c-4b4f-b332-d12657e20737 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:05.781331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657a460e-230c-4b4f-b332-d12657e20737 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:05.784163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657a460e-230c-4b4f-b332-d12657e20737 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:05.813630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657a460e-230c-4b4f-b332-d12657e20737 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:05.818148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657a460e-230c-4b4f-b332-d12657e20737 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:05.832021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1103/2217] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:06.659585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:06.661138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:06.680327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d00b0007-05cb-49de-9041-28534fde3ba3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:06.688172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:06.688838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:06.689074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:06.689346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] PATH_INFO: `/v3/projects/020f8f15edbc422e80de73875ff2786f` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:06.701121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] RBAC: Authorizing `identity:delete_project(project_id=020f8f15edbc422e80de73875ff2786f)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:06.703868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:06.976102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '020f8f15edbc422e80de73875ff2786f', 'request_id': 'req-487b93de-5c02-4a6d-b9c8-c80c955da22a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:07.085450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '020f8f15edbc422e80de73875ff2786f', 'request_id': 'req-487b93de-5c02-4a6d-b9c8-c80c955da22a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:07.092724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '020f8f15edbc422e80de73875ff2786f', 'request_id': 'req-487b93de-5c02-4a6d-b9c8-c80c955da22a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:07.103146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] The token cache is being invalidate because project 020f8f15edbc422e80de73875ff2786f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:07.103398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:07.104058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-487b93de-5c02-4a6d-b9c8-c80c955da22a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:07.106362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1115/2218] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:06 2022] DELETE /identity/v3/projects/020f8f15edbc422e80de73875ff2786f => generated 0 bytes in 448 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:07.112604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.113177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.113273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.136558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:07.137854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:07.202172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1104/2219] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:07.210945 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.211442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.211702 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.232614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:07.233737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:07.241581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1116/2220] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:07.246425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:07.256309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14cbc3b9-4cb6-4054-9ef8-21bc2860d0bc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:07.261170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:07.261856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.262134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.262408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.263139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:38:07.263464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:07.265056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:07.277821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1105/2221] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:38:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:38:07.282044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:07.291814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9705e9f-7d2c-4bdc-ae13-387eed227ebb None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:07.297362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:07.297961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.298237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.298548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.299916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:07.302762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:07.454315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1117/2222] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:07 2022] POST /identity/v3/projects => generated 373 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:07.455031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:07.467326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-120218c0-ea96-44be-ab3e-0442d9e3bb9a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:07.549361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:07.550036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.550316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.550587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.551373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:07.553473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:07.758003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1106/2223] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:07 2022] POST /identity/v3/users => generated 287 bytes in 303 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:38:07.758900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:07.771741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d854c1d8-9219-4643-8120-b915d9579c1f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:07.784128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:07.784961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.785243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.785764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.786651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:07.788624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:07.799791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1118/2224] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:07 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:38:07.803806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:07.814201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193fb043-793e-4220-a3a0-a614923f5de0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:07.818986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ba82a35-0492-4ddb-8ca6-80397f07ecb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:07.819601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba82a35-0492-4ddb-8ca6-80397f07ecb5 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.819865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba82a35-0492-4ddb-8ca6-80397f07ecb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.820134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ba82a35-0492-4ddb-8ca6-80397f07ecb5 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.820795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba82a35-0492-4ddb-8ca6-80397f07ecb5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:07.822532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ba82a35-0492-4ddb-8ca6-80397f07ecb5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:07.833400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1107/2225] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:07.837538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:07.848048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bada70f-c120-4c8c-8cc3-2b15cef46a9c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:07.852501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:07.853136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.853411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.853679 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] PATH_INFO: `/v3/projects/54a4b85dba3b48b68e25c190b98b6197/users/f7e32dbc314d4953b3e3c21cbb481367/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:07.867914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] RBAC: Authorizing `identity:create_grant(project_id=54a4b85dba3b48b68e25c190b98b6197, user_id=f7e32dbc314d4953b3e3c21cbb481367, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:07.871372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:07.974540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1119/2226] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:07 2022] PUT /identity/v3/projects/54a4b85dba3b48b68e25c190b98b6197/users/f7e32dbc314d4953b3e3c21cbb481367/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:07.979627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:07.979915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:07.980192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:08.005551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:08.007669 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:08.030144 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:08.071640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1108/2227] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:08.082142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:08.083937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:08.098262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:08.143664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3800ad2-68bc-4f43-8b49-844a3606e9aa admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:08.147629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e93192-06ab-451e-a4b2-234e03ac237b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:08.148352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e93192-06ab-451e-a4b2-234e03ac237b service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:08.148637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e93192-06ab-451e-a4b2-234e03ac237b service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:08.148903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e93192-06ab-451e-a4b2-234e03ac237b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:08.158866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e93192-06ab-451e-a4b2-234e03ac237b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:08.160475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e93192-06ab-451e-a4b2-234e03ac237b service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:08.173282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1120/2228] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:08.395241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:08.397274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:08.415216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d5cff27-58c8-43ef-9329-c2408e181702 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:08.423837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:08.424492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:08.424775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:08.425077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] PATH_INFO: `/v3/users/9031384442d04ecf84b98138bb10f359` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:08.426273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] RBAC: Authorizing `identity:delete_user(user_id=9031384442d04ecf84b98138bb10f359)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:08.428267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:08.665402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9031384442d04ecf84b98138bb10f359'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:08.682997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9031384442d04ecf84b98138bb10f359'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:08.683803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9031384442d04ecf84b98138bb10f359'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:08.684534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9031384442d04ecf84b98138bb10f359'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:08.782135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9031384442d04ecf84b98138bb10f359'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:08.848736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1109/2229] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:38:08 2022] DELETE /identity/v3/users/9031384442d04ecf84b98138bb10f359 => generated 0 bytes in 454 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:08.856542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:08.857630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:08.858142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:08.880784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:08.882028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:08.976958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1121/2230] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:08.988905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:08.990486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:09.067208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62f59f33-4888-460c-9c31-53b5dbb43dcd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:09.071160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b769a5a5-6818-409e-a2a9-8fd53e6add3b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:09.071924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b769a5a5-6818-409e-a2a9-8fd53e6add3b service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:09.072201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b769a5a5-6818-409e-a2a9-8fd53e6add3b service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:09.072471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b769a5a5-6818-409e-a2a9-8fd53e6add3b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:09.084659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b769a5a5-6818-409e-a2a9-8fd53e6add3b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:09.086394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b769a5a5-6818-409e-a2a9-8fd53e6add3b service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:09.101460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1110/2231] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:09.703834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:09.706346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:09.713117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:09.714571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:09.715240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:09.725139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-959c3b38-8aad-4d19-b7d3-a4a54eb99b2f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:09.730528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:09.737844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:09.737844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:09.737844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] PATH_INFO: `/v3/projects/a8ff4b44d04942c4b2599eba4b0dafa6` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:09.737844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8ff4b44d04942c4b2599eba4b0dafa6)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:09.737844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:09.766867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] MFA Rules not processed for user `f7e32dbc314d4953b3e3c21cbb481367`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:09.766867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:09.814663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:09.858931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1111/2232] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:09.881985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:09.885725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:09.930746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8ff4b44d04942c4b2599eba4b0dafa6', 'request_id': 'req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:09.956086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a655fd91-68fc-4f34-9f3a-3c2d7906f382 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.000961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8ff4b44d04942c4b2599eba4b0dafa6', 'request_id': 'req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.009247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8ff4b44d04942c4b2599eba4b0dafa6', 'request_id': 'req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.019901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.020623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.020938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.021231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.023176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] The token cache is being invalidate because project a8ff4b44d04942c4b2599eba4b0dafa6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:10.023655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.024299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-a61712c3-47c6-4ec5-b7c5-3cfe96b893fc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.025863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1122/2233] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:09 2022] DELETE /identity/v3/projects/a8ff4b44d04942c4b2599eba4b0dafa6 => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:10.032033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97678957-31cf-49fd-9759-521452625794 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.032472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97678957-31cf-49fd-9759-521452625794 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.033116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97678957-31cf-49fd-9759-521452625794 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.055829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-97678957-31cf-49fd-9759-521452625794 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:10.057191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-97678957-31cf-49fd-9759-521452625794 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:10.097190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.098861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.100080 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:10.132373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1123/2234] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:10.136720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1112/2235] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:09 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 255 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:10.143431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.143431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.143431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.149922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.150653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:10.178321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e59299cb-b704-4ebd-aa21-a832c64a3648 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.184817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.185284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:10.185461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.185885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.186166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] PATH_INFO: `/v3/users/addc1913f3324438aa9d3a626e7fae70` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.188072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] RBAC: Authorizing `identity:delete_user(user_id=addc1913f3324438aa9d3a626e7fae70)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.190129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.190508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:10.220049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1124/2236] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:10.223725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.241135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03b86a2b-b424-4edb-bbe0-3c1340002a09 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.247732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.253114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.253114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.253114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.253114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:38:10.253114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.253114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.268430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1125/2237] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:38:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:38:10.274669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.286287 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8b3bd19-ea4a-4648-9e1f-0d47d65ac89a admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.302424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.303087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.303358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.303674 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.304256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.306327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.408937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:10.417448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1126/2238] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:10 2022] POST /identity/v3/projects => generated 383 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:10.419351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.422324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'addc1913f3324438aa9d3a626e7fae70'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.430660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'addc1913f3324438aa9d3a626e7fae70'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.431438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'addc1913f3324438aa9d3a626e7fae70'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.432130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'addc1913f3324438aa9d3a626e7fae70'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.481288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76500550-5473-470d-a165-f585357af5d3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.485961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.486560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.486868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.487137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.487766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.489685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.526397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c9705307-e824-410e-89f7-f612370e287f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'addc1913f3324438aa9d3a626e7fae70'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:10.552595 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1113/2239] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:38:10 2022] DELETE /identity/v3/users/addc1913f3324438aa9d3a626e7fae70 => generated 0 bytes in 406 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:10.554642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.554949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.555226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.582739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:10.584027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:10.643930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1127/2240] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:10 2022] POST /identity/v3/users => generated 292 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:10.646393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.648222 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:10.655530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1114/2241] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:10.663908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14b714ca-6875-4eda-9c59-ac60ecc9e524 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.672794 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.673276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.673814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.675506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.677350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:10.684196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.684196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.685431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.702625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1128/2242] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:10 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:10.706839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.719179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab1a31c-d0ac-4f01-8819-1a21c37bb1d3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.724258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.724934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.725278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.725623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.726331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.728035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.740187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1129/2243] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:10 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:10.748497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:10.751227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6651e988-e81b-41be-9ef1-9e7d093a5fc9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.755160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed6a194a-13a1-44a6-8c57-dc37b71f458e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.755883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a194a-13a1-44a6-8c57-dc37b71f458e service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.756236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a194a-13a1-44a6-8c57-dc37b71f458e service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.756603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed6a194a-13a1-44a6-8c57-dc37b71f458e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.759712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edfe3b6-8b29-4a51-b2f1-644837e332cf admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:10.765488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:10.766060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.766410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.766656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] PATH_INFO: `/v3/projects/d5fb5144fbf64abe8ec6b37c7f0444b3/users/978e8037ce1d4742abe66ea12d660479/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.769206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6a194a-13a1-44a6-8c57-dc37b71f458e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.770806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed6a194a-13a1-44a6-8c57-dc37b71f458e service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.781455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1115/2244] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:10 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:10.782527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] RBAC: Authorizing `identity:create_grant(project_id=d5fb5144fbf64abe8ec6b37c7f0444b3, user_id=978e8037ce1d4742abe66ea12d660479, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:10.785436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:10.922805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1130/2245] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:10 2022] PUT /identity/v3/projects/d5fb5144fbf64abe8ec6b37c7f0444b3/users/978e8037ce1d4742abe66ea12d660479/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:38:10.926922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:10.927224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:10.927468 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:10.948190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:10.951361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:11.080905 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1116/2246] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:10 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:11.116859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:11.225413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:11.243544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20cc11cc-dfd4-4ebf-9daa-a21d1488b94b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:11.249204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aab0758d-a6d4-4af3-b4e0-9a44176d3432 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:11.249907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab0758d-a6d4-4af3-b4e0-9a44176d3432 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:11.250191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab0758d-a6d4-4af3-b4e0-9a44176d3432 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:11.250588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aab0758d-a6d4-4af3-b4e0-9a44176d3432 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:11.262575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab0758d-a6d4-4af3-b4e0-9a44176d3432 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:11.264237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aab0758d-a6d4-4af3-b4e0-9a44176d3432 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:11.278708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1131/2247] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 162 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:11.280267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:11.425697 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:11.427328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:11.447828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac83b8c4-7c02-4b03-902b-54d003d8a0a5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:11.468342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:11.469108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:11.469681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:11.469794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] PATH_INFO: `/v3/projects/3e379e32c618473bbe866b18cc1e81c3` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:11.471262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e379e32c618473bbe866b18cc1e81c3)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:11.473385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:11.644391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e379e32c618473bbe866b18cc1e81c3', 'request_id': 'req-e2986d30-7a9b-4c82-99cc-296390a2a598'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:11.653650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e379e32c618473bbe866b18cc1e81c3', 'request_id': 'req-e2986d30-7a9b-4c82-99cc-296390a2a598'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:11.778037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e379e32c618473bbe866b18cc1e81c3', 'request_id': 'req-e2986d30-7a9b-4c82-99cc-296390a2a598'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:11.795919 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] The token cache is being invalidate because project 3e379e32c618473bbe866b18cc1e81c3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:11.796251 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:11.796937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e2986d30-7a9b-4c82-99cc-296390a2a598 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:11.798586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1117/2248] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:11 2022] DELETE /identity/v3/projects/3e379e32c618473bbe866b18cc1e81c3 => generated 0 bytes in 373 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:11.803590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:11.803876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:11.804154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:11.825911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:11.827216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:11.917105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1132/2249] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:11.928691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:11.928691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:11.928691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:11.948648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:11.950453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:11.959285 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1118/2250] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:11.962501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:11.972247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacac381-a3c6-4201-aa96-4b9be63cd9a9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:11.989932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:11.996441 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:11.996733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:11.997011 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:11.997686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:38:11.998013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:11.999661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2386a8b-7f00-4700-9ecd-18d6c1967e43 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:12.029318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1133/2251] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:38:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 67 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:38:12.050804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:12.055465 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-948bfbe3-4ad8-47e3-872f-a32768bf05b9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:12.061628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:12.062243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.062521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.062800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.063449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:12.065425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:12.179904 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.180405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.180405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.192937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1119/2252] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:12 2022] POST /identity/v3/projects => generated 399 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:12.197297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:12.200912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] MFA Rules not processed for user `f7e32dbc314d4953b3e3c21cbb481367`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:12.202099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:12.278895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1134/2253] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:12 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:12.297346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c875672-ef65-48f1-b1eb-381943987cb5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:12.302933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:12.303616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.303949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.304235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.304875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:12.306917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:12.419584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1120/2254] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:12 2022] POST /identity/v3/users => generated 300 bytes in 223 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:38:12.423822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:12.435498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11b6b485-a796-475f-85f7-b5d6206bba45 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:12.441133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:12.441808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.442091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.442362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.443116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:12.444786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:12.455523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1135/2255] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:12.461809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:12.477571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd96072a-8454-4722-805a-28f7cacc6a1e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:12.486729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2432fe61-bd98-4579-8ed2-32085ebc5060 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:12.487943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2432fe61-bd98-4579-8ed2-32085ebc5060 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.488255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2432fe61-bd98-4579-8ed2-32085ebc5060 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.489093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2432fe61-bd98-4579-8ed2-32085ebc5060 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.489467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2432fe61-bd98-4579-8ed2-32085ebc5060 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:12.491723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2432fe61-bd98-4579-8ed2-32085ebc5060 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:12.504486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1121/2256] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:12 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:12.508248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:12.519466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27379440-d53f-4853-b5f8-0d8d7789edcc admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:12.525193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010e249-2654-45a2-808d-198670796312 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:12.525709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010e249-2654-45a2-808d-198670796312 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.526876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010e249-2654-45a2-808d-198670796312 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.527294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b010e249-2654-45a2-808d-198670796312 admin admin] PATH_INFO: `/v3/projects/ae14322e01ed4efb872fe7db0bb59eb2/users/0edef4e3f2af42eeb9d96b78a788ec9c/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.545781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b010e249-2654-45a2-808d-198670796312 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae14322e01ed4efb872fe7db0bb59eb2, user_id=0edef4e3f2af42eeb9d96b78a788ec9c, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:12.548709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b010e249-2654-45a2-808d-198670796312 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:12.630006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1136/2257] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:12 2022] PUT /identity/v3/projects/ae14322e01ed4efb872fe7db0bb59eb2/users/0edef4e3f2af42eeb9d96b78a788ec9c/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:38:12.650498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4841ee-0c4e-41e8-ac29-b7599cb8046f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.651916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4841ee-0c4e-41e8-ac29-b7599cb8046f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.652362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb4841ee-0c4e-41e8-ac29-b7599cb8046f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.727644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fb4841ee-0c4e-41e8-ac29-b7599cb8046f None None] MFA Rules not processed for user `0edef4e3f2af42eeb9d96b78a788ec9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:12.731598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fb4841ee-0c4e-41e8-ac29-b7599cb8046f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:12.760970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:12.863320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1122/2258] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:12 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:12.878155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010e249-2654-45a2-808d-198670796312 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:12.880834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b010e249-2654-45a2-808d-198670796312 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:12.947056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:12.965848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b010e249-2654-45a2-808d-198670796312 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:12.971358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:12.972189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:12.973341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:12.973341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:12.987739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:12.989339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:13.005866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1137/2259] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:12 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:38:18.012539 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.013022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.013116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.045721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] MFA Rules not processed for user `978e8037ce1d4742abe66ea12d660479`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:18.047108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:18.141823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1123/2260] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:18 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:18.146157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:18.147656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:18.238494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e0d925b-e7f5-4dd5-b53d-1fa0ae5a5b29 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:18.245956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:18.245956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.245956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.245956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.249040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:18.249415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:18.370154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1138/2261] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:18 2022] POST /identity/v3/projects => generated 385 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:38:18.374415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:18.386438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19bc588e-a982-40e5-be1f-5a17b68f71cd None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:18.458725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:18.459508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.459855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.460137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.460960 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:18.463339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:18.551351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1124/2262] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:18 2022] POST /identity/v3/users => generated 299 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:18.555752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:18.567565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e35d5893-a131-4160-955b-c5640937bc9d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:18.572442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:18.588009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.588009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.588009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.588009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:18.588009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:18.603819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:18.604556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1139/2263] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:18.617336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a03bcd2b-4860-4147-a13f-dbb4009511aa admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:18.631363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:18.632711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.633296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.633779 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] PATH_INFO: `/v3/projects/adcfb47edc514af082ac07df9bfc3dbc/users/883ebd32de2e4cd9a21df28b7297e100/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.661504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adcfb47edc514af082ac07df9bfc3dbc, user_id=883ebd32de2e4cd9a21df28b7297e100, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:18.664218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:18.771542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1125/2264] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:18 2022] PUT /identity/v3/projects/adcfb47edc514af082ac07df9bfc3dbc/users/883ebd32de2e4cd9a21df28b7297e100/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:18.772497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:18.783678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43c81303-be7c-4eff-a664-9aecd1e989dc admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:18.850397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:18.851140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.851457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.851997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.852606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:18.860055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:18.871522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1140/2265] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:18 2022] GET /identity/v3/roles => generated 1420 bytes in 96 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:18.872875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:18.885082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c01911c-6b35-4153-928d-97ab652ac092 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:18.893924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22bcda3e-1212-4e02-9dd0-b2ef80e9f092 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:18.894611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bcda3e-1212-4e02-9dd0-b2ef80e9f092 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:18.895016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bcda3e-1212-4e02-9dd0-b2ef80e9f092 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:18.895438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22bcda3e-1212-4e02-9dd0-b2ef80e9f092 admin admin] PATH_INFO: `/v3/projects/adcfb47edc514af082ac07df9bfc3dbc/users/883ebd32de2e4cd9a21df28b7297e100/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:18.899930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bcda3e-1212-4e02-9dd0-b2ef80e9f092 admin admin] RBAC: Authorizing `identity:create_grant(project_id=adcfb47edc514af082ac07df9bfc3dbc, user_id=883ebd32de2e4cd9a21df28b7297e100, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:18.903418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22bcda3e-1212-4e02-9dd0-b2ef80e9f092 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:18.911498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:18.916690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1126/2266] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:18 2022] PUT /identity/v3/projects/adcfb47edc514af082ac07df9bfc3dbc/users/883ebd32de2e4cd9a21df28b7297e100/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 44 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:22.602307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:22.627694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c7bd61-b35d-459b-b366-ef5af66df1a7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:22.740142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:22.751820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:22.752596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:22.752886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:22.754258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] PATH_INFO: `/v3/users/0edef4e3f2af42eeb9d96b78a788ec9c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:22.755210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] RBAC: Authorizing `identity:delete_user(user_id=0edef4e3f2af42eeb9d96b78a788ec9c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:22.758420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:23.100660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0edef4e3f2af42eeb9d96b78a788ec9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:23.102028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0edef4e3f2af42eeb9d96b78a788ec9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:23.122876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0edef4e3f2af42eeb9d96b78a788ec9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:23.130493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0edef4e3f2af42eeb9d96b78a788ec9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:23.215854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0edef4e3f2af42eeb9d96b78a788ec9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:23.251727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1141/2267] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:38:22 2022] DELETE /identity/v3/users/0edef4e3f2af42eeb9d96b78a788ec9c => generated 0 bytes in 647 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:23.257366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:23.259467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:23.259959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:23.300981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:23.303866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:23.560000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1127/2268] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 304 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:23.611595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:23.613210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:23.760403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c446efe-b4cc-431d-86c5-f9b1201113ef admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:23.765735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2291c20-fca7-4a25-8c9c-32d2f6ae709a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:23.766615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2291c20-fca7-4a25-8c9c-32d2f6ae709a service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:23.767134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2291c20-fca7-4a25-8c9c-32d2f6ae709a service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:23.767706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2291c20-fca7-4a25-8c9c-32d2f6ae709a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:23.797551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2291c20-fca7-4a25-8c9c-32d2f6ae709a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:23.799068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2291c20-fca7-4a25-8c9c-32d2f6ae709a service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:23.823714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1142/2269] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 212 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:24.988009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:24.989469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:25.004359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d853bcc-4595-4132-ba3c-b3cb7482599a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:25.011261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:25.016284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:25.016284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:25.016284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] PATH_INFO: `/v3/projects/ae14322e01ed4efb872fe7db0bb59eb2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:25.016284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae14322e01ed4efb872fe7db0bb59eb2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:25.016284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:25.150123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae14322e01ed4efb872fe7db0bb59eb2', 'request_id': 'req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:25.162035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae14322e01ed4efb872fe7db0bb59eb2', 'request_id': 'req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:25.231490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae14322e01ed4efb872fe7db0bb59eb2', 'request_id': 'req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:25.253548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] The token cache is being invalidate because project ae14322e01ed4efb872fe7db0bb59eb2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:25.253780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:25.254496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:25.255985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1128/2270] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:24 2022] DELETE /identity/v3/projects/ae14322e01ed4efb872fe7db0bb59eb2 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:27.378076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:27.378626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:27.378722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:27.409300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] MFA Rules not processed for user `883ebd32de2e4cd9a21df28b7297e100`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:27.411524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:27.507782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1143/2271] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:27 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:27.516777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:27.521126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:27.595186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c19ab02-c047-46b8-9dcb-0b3bec0b6b1c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:27.601271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:27.602086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:27.602367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:27.602640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:27.605504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:27.652526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:27.655134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:27.660478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:27.664628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:27.685721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:27.688362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed0f9290-b9d2-4882-9aeb-ac8b7b4f2ee0 service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:27.708238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1129/2272] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:27 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:38:27.739032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6282359a-3b8c-49ca-b70f-0e3d2caa5ed6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:27.744184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:27.745448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:27.745770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:27.746117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] PATH_INFO: `/v3/users/379c7b5e94c842b3b757450b345de820` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:27.747513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] RBAC: Authorizing `identity:delete_user(user_id=379c7b5e94c842b3b757450b345de820)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:27.749961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:28.010928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '379c7b5e94c842b3b757450b345de820'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:28.011592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '379c7b5e94c842b3b757450b345de820'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:28.018913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '379c7b5e94c842b3b757450b345de820'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:28.019648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '379c7b5e94c842b3b757450b345de820'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:28.102170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:28.106844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:28.106844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:28.123390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:28.124813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:28.151583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-41e64d6c-d76f-4704-af76-c7a18ab2dd6e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '379c7b5e94c842b3b757450b345de820'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:28.194270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1144/2273] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:38:27 2022] DELETE /identity/v3/users/379c7b5e94c842b3b757450b345de820 => generated 0 bytes in 542 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:38:28.199521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:28.200010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:28.204260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:28.209865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1130/2274] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:38:28 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:28.226558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:28.227938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:28.267873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:28.269462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:28.299479 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1145/2275] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:28 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:28.308418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:28.309883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:28.372058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f7222d-29de-4653-bc46-f2b0f97f7dc5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:28.388591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf1947f-d7c8-4856-8e7a-7ef820f17a53 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:28.397174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:28.399271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:28.399559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:28.399833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:28.413088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:28.414548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:28.428481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1146/2276] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:28 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:28.464900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b96d7198-bc3d-4c18-9921-09bccae96c92 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:28.465525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96d7198-bc3d-4c18-9921-09bccae96c92 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:28.465812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96d7198-bc3d-4c18-9921-09bccae96c92 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:28.466121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b96d7198-bc3d-4c18-9921-09bccae96c92 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:28.658509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96d7198-bc3d-4c18-9921-09bccae96c92 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:28.659717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b96d7198-bc3d-4c18-9921-09bccae96c92 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:28.676032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1131/2277] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:28 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:38:28.720278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:28.790042 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:29.200063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:29.203014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:29.229654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aebb39f9-3ed3-4ee6-9202-1f0a8b9ec92c service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:29.245065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:29.247311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:29.247591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:29.247861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] PATH_INFO: `/v3/projects/b4ef72b5c4db448293a748cdeace41d0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:29.257583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b4ef72b5c4db448293a748cdeace41d0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:29.259547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:29.394617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b4ef72b5c4db448293a748cdeace41d0', 'request_id': 'req-bee91c0f-d27c-4d4c-9ff7-a87da135a775'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:29.468644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b4ef72b5c4db448293a748cdeace41d0', 'request_id': 'req-bee91c0f-d27c-4d4c-9ff7-a87da135a775'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:29.480648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b4ef72b5c4db448293a748cdeace41d0', 'request_id': 'req-bee91c0f-d27c-4d4c-9ff7-a87da135a775'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:29.551991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] The token cache is being invalidate because project b4ef72b5c4db448293a748cdeace41d0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:29.551991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:29.552720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-bee91c0f-d27c-4d4c-9ff7-a87da135a775 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:29.572122 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1147/2278] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:29 2022] DELETE /identity/v3/projects/b4ef72b5c4db448293a748cdeace41d0 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:29.572830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:29.573324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:29.573749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:29.608143 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:29.612349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:29.803309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:29.803624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:29.804077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:29.810073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1132/2279] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 237 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:29.831692 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:29.834333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:29.857061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1148/2280] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:29.861409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:29.875990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebed7c8-1e5a-4b64-8c8f-f01c1be2941e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:29.893362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:29.894041 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:29.894323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:29.894633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:29.895346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:38:29.895858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:29.897929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:30.094142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1133/2281] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:38:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 76 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:38:30.094462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:30.094462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-717367ef-01c0-48cf-9af3-0a825191fe11 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:30.094462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:30.094462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:30.094857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:30.094857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:30.094857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:30.094857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:30.697923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1149/2282] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:29 2022] POST /identity/v3/projects => generated 381 bytes in 759 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:30.704719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:30.719747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102a0b8-9b5b-4012-95f3-f3215a63942b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:30.913364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:30.920566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:30.921158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:30.921493 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:30.922957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:30.925292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:31.016625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1134/2283] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:30 2022] POST /identity/v3/users => generated 291 bytes in 312 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:31.021065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:31.036846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6583cc9-3b88-4b9f-b7ae-c1101367528f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:31.061215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:31.061347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:31.061662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:31.061938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:31.066959 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:31.071118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:31.118235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1150/2284] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:31 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:31.118443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:31.132479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e2f1c8-29bc-46a7-b837-712eb5b8ae9b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:31.148911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9321b7e-e4f0-4ee8-9bb0-d032eac30149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:31.149535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9321b7e-e4f0-4ee8-9bb0-d032eac30149 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:31.149810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9321b7e-e4f0-4ee8-9bb0-d032eac30149 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:31.150079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9321b7e-e4f0-4ee8-9bb0-d032eac30149 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:31.151511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9321b7e-e4f0-4ee8-9bb0-d032eac30149 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:31.167037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9321b7e-e4f0-4ee8-9bb0-d032eac30149 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:31.205624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1135/2285] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:31 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:31.207914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:31.239024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cce6bd6-f0ef-4119-8343-c682d0f03420 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:31.248948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:31.249921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:31.250290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:31.250659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] PATH_INFO: `/v3/projects/a06947e0733946bb8e266382c0d4d0ce/users/211efc6d124344c0b6ba83c378425df0/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:31.337603 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a06947e0733946bb8e266382c0d4d0ce, user_id=211efc6d124344c0b6ba83c378425df0, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:31.349887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:31.467677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1151/2286] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:31 2022] PUT /identity/v3/projects/a06947e0733946bb8e266382c0d4d0ce/users/211efc6d124344c0b6ba83c378425df0/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 260 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:31.473009 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ead77b2-aa5e-45e1-8fb5-4c7237b88439 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:31.473294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ead77b2-aa5e-45e1-8fb5-4c7237b88439 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:31.473567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ead77b2-aa5e-45e1-8fb5-4c7237b88439 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:31.524001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3ead77b2-aa5e-45e1-8fb5-4c7237b88439 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:31.534631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3ead77b2-aa5e-45e1-8fb5-4c7237b88439 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:31.624332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:31.803055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1136/2287] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 331 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:31.811744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:31.813523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:32.010821 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:32.060727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3891a5f8-f857-48ba-ac4d-a06aa42e0931 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:32.078305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:32.080357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:32.080914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:32.081562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:32.104776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:32.110384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:32.157183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1152/2288] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:31 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:34.829889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed874d09-1a7d-4480-9b78-8d84a0b9b419 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:34.831931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed874d09-1a7d-4480-9b78-8d84a0b9b419 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:34.831931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed874d09-1a7d-4480-9b78-8d84a0b9b419 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:34.855578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ed874d09-1a7d-4480-9b78-8d84a0b9b419 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:34.855727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ed874d09-1a7d-4480-9b78-8d84a0b9b419 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:34.922017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1137/2289] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:38:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:34.957474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:34.959568 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:35.091822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5475bb4-7afb-4b75-8c79-328a4f6197b5 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:35.092501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec778f1-b001-4db2-b340-fa88e6907d06 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:35.092883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec778f1-b001-4db2-b340-fa88e6907d06 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:35.093288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ec778f1-b001-4db2-b340-fa88e6907d06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:35.104526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:35.105506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:35.106337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:35.107193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:35.122651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:35.124654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:35.138895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1153/2290] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 182 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 27 16:38:35.149941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7ec778f1-b001-4db2-b340-fa88e6907d06 None None] MFA Rules not processed for user `211efc6d124344c0b6ba83c378425df0`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:35.152447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7ec778f1-b001-4db2-b340-fa88e6907d06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:35.315861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1138/2291] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:35 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 224 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:35.329003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:35.333450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:35.466845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bad43e2e-7b6d-4eb8-930a-0c87d617c13d service swift] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:35.474883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb5d1a8-190c-4c8e-83a4-7590e4ebc01e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:35.475562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb5d1a8-190c-4c8e-83a4-7590e4ebc01e service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:35.475937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb5d1a8-190c-4c8e-83a4-7590e4ebc01e service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:35.476438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb5d1a8-190c-4c8e-83a4-7590e4ebc01e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:35.496150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb5d1a8-190c-4c8e-83a4-7590e4ebc01e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:35.497551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eb5d1a8-190c-4c8e-83a4-7590e4ebc01e service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:35.525909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1154/2292] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:35 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 197 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:36.530337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:36.530819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:36.533788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:36.576670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] MFA Rules not processed for user `e8d3af2b82b641aea11ebf79677a3af1`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:36.579006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:36.687008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1139/2293] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:36 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:36.833386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bff7629-b02a-40cd-82e0-d012cb9a0659 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:36.834669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bff7629-b02a-40cd-82e0-d012cb9a0659 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:36.834669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bff7629-b02a-40cd-82e0-d012cb9a0659 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:36.861098 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2bff7629-b02a-40cd-82e0-d012cb9a0659 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:36.863229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2bff7629-b02a-40cd-82e0-d012cb9a0659 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:36.873951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1155/2294] 10.222.0.64 () {58 vars in 967 bytes} [Tue Sep 27 16:38:36 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:36.922871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:36.948100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb7b89ce-04de-4e09-bc20-5409984ea6c7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:36.961449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8fd3f9-0d59-4637-8ac9-e17a26988e85 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:36.962193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8fd3f9-0d59-4637-8ac9-e17a26988e85 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:36.962800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8fd3f9-0d59-4637-8ac9-e17a26988e85 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:36.964186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8fd3f9-0d59-4637-8ac9-e17a26988e85 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:36.980637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8fd3f9-0d59-4637-8ac9-e17a26988e85 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:36.983223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8fd3f9-0d59-4637-8ac9-e17a26988e85 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:36.989857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:37.000525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1140/2295] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:36 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:38:37.662581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:37.669662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:37.669662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:37.692074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] MFA Rules not processed for user `211efc6d124344c0b6ba83c378425df0`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:37.693567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:37.711418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1156/2296] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:37 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:37.906104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e1cd9-e95b-46bc-b4f3-b1ad525727c9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:37.907640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e1cd9-e95b-46bc-b4f3-b1ad525727c9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:37.907926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34e1cd9-e95b-46bc-b4f3-b1ad525727c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:37.938513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c34e1cd9-e95b-46bc-b4f3-b1ad525727c9 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:37.958519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c34e1cd9-e95b-46bc-b4f3-b1ad525727c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:37.984286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1141/2297] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:38:37 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:37.999168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:38.017722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd6035a-b67d-462c-ad08-31f977da02a9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:38.023284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:38.024130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:38.024606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:38.025050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:38.036397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:38.039178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:38.049970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1157/2298] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:38:39.459631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bf6af-1d27-42de-a9d2-c5dff41160ff None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:39.460523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bf6af-1d27-42de-a9d2-c5dff41160ff None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:39.460966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-374bf6af-1d27-42de-a9d2-c5dff41160ff None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:39.482081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-374bf6af-1d27-42de-a9d2-c5dff41160ff None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:39.501087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-374bf6af-1d27-42de-a9d2-c5dff41160ff None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:39.501087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1142/2299] 10.222.0.64 () {58 vars in 967 bytes} [Tue Sep 27 16:38:39 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:39.515977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:39.533116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee8e8e56-f73d-487d-aed3-d5f1cc194578 service swift] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:39.549425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:39.550325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:39.550846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:39.551331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:39.570881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:39.575937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:39.593306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1158/2300] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:39 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 78 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:38:42.663527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:42.664034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:42.664116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:42.701639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:42.704896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:42.747850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1143/2301] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:38:42 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:42.774696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:42.793641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-835f2868-16df-47bf-8beb-74410e38d974 service swift] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:42.812602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a7713b1-29c5-44a7-b662-2b0ac229475a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:42.813188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7713b1-29c5-44a7-b662-2b0ac229475a service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:42.813463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7713b1-29c5-44a7-b662-2b0ac229475a service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:42.813733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a7713b1-29c5-44a7-b662-2b0ac229475a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:42.852744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7713b1-29c5-44a7-b662-2b0ac229475a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:42.854745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a7713b1-29c5-44a7-b662-2b0ac229475a service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:42.889678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1159/2302] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:38:43.469977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:43.469977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:43.540705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723134dc-d55e-4833-8d0f-73b3b2465865 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:43.542640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723134dc-d55e-4833-8d0f-73b3b2465865 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:43.543473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-723134dc-d55e-4833-8d0f-73b3b2465865 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:43.562022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fec544a-f8e8-45d7-99f1-1917c2bfc87e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:43.582567 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:43.585303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:43.585303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:43.585303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] PATH_INFO: `/v3/users/e1036377d4d94bb3a929aee17ca5eeec` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:43.585818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1036377d4d94bb3a929aee17ca5eeec)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:43.588102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:43.600212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-723134dc-d55e-4833-8d0f-73b3b2465865 None None] MFA Rules not processed for user `f7e32dbc314d4953b3e3c21cbb481367`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:43.613890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-723134dc-d55e-4833-8d0f-73b3b2465865 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:43.707914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:43.870407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1160/2303] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:43 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 326 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:44.473026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1036377d4d94bb3a929aee17ca5eeec'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:44.497322 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1036377d4d94bb3a929aee17ca5eeec'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:44.498626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1036377d4d94bb3a929aee17ca5eeec'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:44.503819 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1036377d4d94bb3a929aee17ca5eeec'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:44.642303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1036377d4d94bb3a929aee17ca5eeec'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:44.682216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:44.687871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1144/2304] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:38:43 2022] DELETE /identity/v3/users/e1036377d4d94bb3a929aee17ca5eeec => generated 0 bytes in 1230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:38:44.688016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:44.688016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:44.741805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:44.743419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:44.915144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1161/2305] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:44.955438 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:44.956958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:45.151147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bf710a5-6acc-43b9-ad0f-bb5824dda0e6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:45.156366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c0b2a98-b96e-438d-a116-293d6ea7e486 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:45.157855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0b2a98-b96e-438d-a116-293d6ea7e486 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:45.159122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0b2a98-b96e-438d-a116-293d6ea7e486 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:45.160540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c0b2a98-b96e-438d-a116-293d6ea7e486 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:45.189825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0b2a98-b96e-438d-a116-293d6ea7e486 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:45.191356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c0b2a98-b96e-438d-a116-293d6ea7e486 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:45.215174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1145/2306] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:45.912811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:45.914282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:45.932833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e20ef57-9cc7-49ea-9e31-72a2896b3a6b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:45.943345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:45.944020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:45.945648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:45.946092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] PATH_INFO: `/v3/projects/808169451dd64aa1a6a656ed9abbe5a0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:45.947915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] RBAC: Authorizing `identity:delete_project(project_id=808169451dd64aa1a6a656ed9abbe5a0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:45.950552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:46.122547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '808169451dd64aa1a6a656ed9abbe5a0', 'request_id': 'req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:46.253765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '808169451dd64aa1a6a656ed9abbe5a0', 'request_id': 'req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:46.263630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '808169451dd64aa1a6a656ed9abbe5a0', 'request_id': 'req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:46.283831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] The token cache is being invalidate because project 808169451dd64aa1a6a656ed9abbe5a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:38:46.284280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:46.288470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ceb3e901-c9f4-44e1-bbb8-9b1b3dea0e2b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:38:46.295160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1162/2307] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:38:45 2022] DELETE /identity/v3/projects/808169451dd64aa1a6a656ed9abbe5a0 => generated 0 bytes in 379 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:38:46.297910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec5d289-09de-4d15-b172-7127634550f8 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:46.298461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec5d289-09de-4d15-b172-7127634550f8 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:46.298556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eec5d289-09de-4d15-b172-7127634550f8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:46.354830 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-eec5d289-09de-4d15-b172-7127634550f8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:46.360608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eec5d289-09de-4d15-b172-7127634550f8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:46.437194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:46.465691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:46.466673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:46.467398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:46.493462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1146/2308] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:46.497436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:46.498811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:46.499927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:46.528322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] MFA Rules not processed for user `978e8037ce1d4742abe66ea12d660479`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:46.532817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:46.545659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:46.547474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:46.559461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1147/2309] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:46 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 65 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:46.566997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:46.580459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a5e8c53-e68a-420b-8ba8-01438c3c66ef None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:46.595633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:46.597253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:46.599236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:46.599523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:46.602643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:38:46.609422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:46.609422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:46.654423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1148/2310] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:38:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:38:46.657278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:46.675896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:38:46.688076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37ac72a-d38a-400e-9ac4-3aa495fb5486 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:46.694955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1163/2311] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:46 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 228 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:46.701001 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:46.701837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:46.702281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:46.702914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:46.703857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:46.706113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:46.846169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1149/2312] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:38:46 2022] POST /identity/v3/projects => generated 377 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:46.847107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:46.864431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f30951-508c-400d-97dc-2e1be51b0386 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:46.939158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:46.939814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:46.940041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:46.940313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:46.940955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:46.942997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:47.060662 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1164/2313] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:38:46 2022] POST /identity/v3/users => generated 289 bytes in 214 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:38:47.065838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:47.079538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faaf0239-7184-4abc-9c00-2cf0eecc6fa7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:47.085334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:47.086033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:47.086305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:47.086645 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:47.087530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:47.092762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:47.114477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1150/2314] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:47 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:38:47.115606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:47.128241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e366cb-a9ad-47f7-a18d-db51537c6e2a admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:47.133219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fa196fa-0ff5-4ac6-9330-3de19d305d78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:47.133972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa196fa-0ff5-4ac6-9330-3de19d305d78 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:47.134504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa196fa-0ff5-4ac6-9330-3de19d305d78 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:47.135010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fa196fa-0ff5-4ac6-9330-3de19d305d78 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:47.141563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa196fa-0ff5-4ac6-9330-3de19d305d78 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:47.141563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fa196fa-0ff5-4ac6-9330-3de19d305d78 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:47.153647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1165/2315] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:38:47 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:38:47.154600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:47.167395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20ad0596-101c-431e-a899-d42ede15bbd7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:47.172941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:47.178813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:47.178813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:47.178813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] PATH_INFO: `/v3/projects/1e0f894b53284e788cdb4864acd917be/users/421aaa238a3c495096dfc55aba76267c/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:47.191999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] RBAC: Authorizing `identity:create_grant(project_id=1e0f894b53284e788cdb4864acd917be, user_id=421aaa238a3c495096dfc55aba76267c, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:47.195398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:47.286218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1151/2316] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:38:47 2022] PUT /identity/v3/projects/1e0f894b53284e788cdb4864acd917be/users/421aaa238a3c495096dfc55aba76267c/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:38:47.297373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ac707-f283-4ff7-b37c-501528a09aab None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:47.297373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ac707-f283-4ff7-b37c-501528a09aab None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:47.297373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-199ac707-f283-4ff7-b37c-501528a09aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:47.316521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-199ac707-f283-4ff7-b37c-501528a09aab None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:47.318258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-199ac707-f283-4ff7-b37c-501528a09aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:47.413590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1166/2317] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:38:47 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:38:47.428871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:47.432916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:47.568863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-334ce37b-cc4e-4547-bfa5-b11d6096dacc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:47.580681 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:47.581336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:47.581625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:47.581921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:47.599880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:47.601685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:47.629240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1152/2318] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:38:47 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 201 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:38:55.751118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af647b0-ee81-462f-a4d3-fd2fb095adb9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:55.751651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af647b0-ee81-462f-a4d3-fd2fb095adb9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:55.751779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3af647b0-ee81-462f-a4d3-fd2fb095adb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:55.802387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3af647b0-ee81-462f-a4d3-fd2fb095adb9 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:38:55.805569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3af647b0-ee81-462f-a4d3-fd2fb095adb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:55.949981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1167/2319] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:38:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:38:56.038033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:38:56.039430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:38:56.196072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a473ce20-ff8a-4cef-955b-f28b639d9c4e service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:38:56.211712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:38:56.213266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:38:56.213777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:38:56.214232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:38:56.235725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:38:56.237604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:38:56.264809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1153/2320] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:38:56 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:39:02.562134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:02.562718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:02.562718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:02.620154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] MFA Rules not processed for user `421aaa238a3c495096dfc55aba76267c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:02.621492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:02.639335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:02.767174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:02.776661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1168/2321] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:39:02 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:02.784406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ecd399a-3f70-4452-a8b2-c9f209aad4f5 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:02.795735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:02.796836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:02.797318 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:02.797783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:02.798592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:02.800728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:02.826752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:03.005299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:03.006451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1154/2322] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:39:02 2022] POST /identity/v3/projects => generated 377 bytes in 240 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:39:03.032450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77256d7c-4222-4969-a495-49e3ed0455f5 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:03.210242 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:03.211338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:03.222392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:03.222952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:03.226154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:03.235280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:03.381088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:03.381728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1169/2323] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:39:03 2022] POST /identity/v3/users => generated 295 bytes in 375 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:39:03.396817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd5bbce0-a00b-4c77-a21a-5d8d02b8c278 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:03.406899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:03.407525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:03.407801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:03.408068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:03.408826 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:03.411628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:03.432304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1155/2324] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:39:03 2022] GET /identity/v3/roles => generated 1420 bytes in 52 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:39:03.436517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:03.448193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8ab18e6-22e8-4968-be87-0adaba75dfb8 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:03.453742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:03.454426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:03.454748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:03.455058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] PATH_INFO: `/v3/projects/2dc58006640a43db85dadea3d504c96b/users/87b67db042cf48a8ab02e3df17950dbb/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:03.480653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dc58006640a43db85dadea3d504c96b, user_id=87b67db042cf48a8ab02e3df17950dbb, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:03.483944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:03.560984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1170/2325] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:39:03 2022] PUT /identity/v3/projects/2dc58006640a43db85dadea3d504c96b/users/87b67db042cf48a8ab02e3df17950dbb/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 124 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:39:03.564667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:03.584824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-031222dc-3e72-43dd-8d95-462d201fc297 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:03.652460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20d1644-3d12-41da-84ff-68e4db1622df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:03.653311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d1644-3d12-41da-84ff-68e4db1622df admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:03.653672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d1644-3d12-41da-84ff-68e4db1622df admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:03.654014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20d1644-3d12-41da-84ff-68e4db1622df admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:03.654890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20d1644-3d12-41da-84ff-68e4db1622df admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:03.656575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20d1644-3d12-41da-84ff-68e4db1622df admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:03.670818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1156/2326] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:39:03 2022] GET /identity/v3/roles => generated 1420 bytes in 107 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:39:03.674665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:03.691497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69b8c710-650b-43d4-ab20-2f70fa606f71 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:03.703803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:03.703803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:03.703803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:03.703803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] PATH_INFO: `/v3/projects/2dc58006640a43db85dadea3d504c96b/users/87b67db042cf48a8ab02e3df17950dbb/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:03.719121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2dc58006640a43db85dadea3d504c96b, user_id=87b67db042cf48a8ab02e3df17950dbb, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:03.726207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:03.751047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1171/2327] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:39:03 2022] PUT /identity/v3/projects/2dc58006640a43db85dadea3d504c96b/users/87b67db042cf48a8ab02e3df17950dbb/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 77 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:39:13.664263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=38,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:13.745327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=65,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:14.800116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70c7920-7fa5-4e36-aae1-2fbb485f4523 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:14.808234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70c7920-7fa5-4e36-aae1-2fbb485f4523 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:14.808234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a70c7920-7fa5-4e36-aae1-2fbb485f4523 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:14.845910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a70c7920-7fa5-4e36-aae1-2fbb485f4523 None None] MFA Rules not processed for user `87b67db042cf48a8ab02e3df17950dbb`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:14.847481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a70c7920-7fa5-4e36-aae1-2fbb485f4523 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:14.942925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1157/2328] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:39:14 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:14.954196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:14.958273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:15.057446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba9d7b25-d45e-44e7-a427-c12d383b05f9 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:15.063200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f881a58-f35a-4ce2-bc6c-cbf9bf08773c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:15.063919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f881a58-f35a-4ce2-bc6c-cbf9bf08773c service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:15.064260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f881a58-f35a-4ce2-bc6c-cbf9bf08773c service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:15.064619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f881a58-f35a-4ce2-bc6c-cbf9bf08773c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:15.153087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f881a58-f35a-4ce2-bc6c-cbf9bf08773c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:15.154806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f881a58-f35a-4ce2-bc6c-cbf9bf08773c service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:15.166974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1172/2329] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:39:14 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 213 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:39:19.380040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973e4b1-e5b6-49ec-968f-49bd8e27fb05 None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:19.380582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973e4b1-e5b6-49ec-968f-49bd8e27fb05 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:19.380582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9973e4b1-e5b6-49ec-968f-49bd8e27fb05 None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:19.381959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1158/2330] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:39:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:39:19.389114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5100961d-235f-4ee8-be52-a40356acc905 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:19.389570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5100961d-235f-4ee8-be52-a40356acc905 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:19.389797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5100961d-235f-4ee8-be52-a40356acc905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:19.421090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5100961d-235f-4ee8-be52-a40356acc905 None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:19.422319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5100961d-235f-4ee8-be52-a40356acc905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:19.434184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1173/2331] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:39:19 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:39:19.446764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973e4b1-e5b6-49ec-968f-49bd8e27fb05 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:19.448672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9973e4b1-e5b6-49ec-968f-49bd8e27fb05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:19.545587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9973e4b1-e5b6-49ec-968f-49bd8e27fb05 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:19.550279 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81a65a6b-3499-4f2e-bba8-2a27d52d3baa service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:19.551363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a65a6b-3499-4f2e-bba8-2a27d52d3baa service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:19.551662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a65a6b-3499-4f2e-bba8-2a27d52d3baa service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:19.551922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81a65a6b-3499-4f2e-bba8-2a27d52d3baa service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:19.566489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a65a6b-3499-4f2e-bba8-2a27d52d3baa service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:19.568488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81a65a6b-3499-4f2e-bba8-2a27d52d3baa service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:19.583345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1159/2332] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:39:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:39:19.795179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603598c-a86b-43d4-9b75-eedef0deb4c6 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:19.795458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603598c-a86b-43d4-9b75-eedef0deb4c6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:19.795722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5603598c-a86b-43d4-9b75-eedef0deb4c6 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:19.797069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1174/2333] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:39:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:39:19.801896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa151c-8390-4bd8-b840-bf971a350c6e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:19.802205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa151c-8390-4bd8-b840-bf971a350c6e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:19.802482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3efa151c-8390-4bd8-b840-bf971a350c6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:19.823803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3efa151c-8390-4bd8-b840-bf971a350c6e None None] MFA Rules not processed for user `28f0f9fef2f54e56a7a92d077c25d2b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:19.825012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3efa151c-8390-4bd8-b840-bf971a350c6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:19.886087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1160/2334] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:39:19 2022] POST /identity/v3/auth/tokens => generated 3121 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:19.898715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603598c-a86b-43d4-9b75-eedef0deb4c6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:19.901133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5603598c-a86b-43d4-9b75-eedef0deb4c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:19.914514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5603598c-a86b-43d4-9b75-eedef0deb4c6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:19.920226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b9e52c5-30e2-4191-86fe-a346a0bb6972 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:19.920837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e52c5-30e2-4191-86fe-a346a0bb6972 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:19.921118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e52c5-30e2-4191-86fe-a346a0bb6972 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:19.921387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b9e52c5-30e2-4191-86fe-a346a0bb6972 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:19.929492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9e52c5-30e2-4191-86fe-a346a0bb6972 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:19.931186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b9e52c5-30e2-4191-86fe-a346a0bb6972 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:19.939534 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1175/2335] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:39:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:39:20.039994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4c892c-5f1e-4df8-a6ce-86e364bdbc9d None None] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:20.040312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4c892c-5f1e-4df8-a6ce-86e364bdbc9d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:20.040556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af4c892c-5f1e-4df8-a6ce-86e364bdbc9d None None] PATH_INFO: `/` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:20.041945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1161/2336] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:39:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:39:20.046883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da241c4-990c-4a2b-93a2-729f684c4afa None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:20.047189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da241c4-990c-4a2b-93a2-729f684c4afa None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:20.047460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6da241c4-990c-4a2b-93a2-729f684c4afa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:20.067950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6da241c4-990c-4a2b-93a2-729f684c4afa None None] MFA Rules not processed for user `ab556e3087544c42929aab1b35d145a4`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:20.069153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6da241c4-990c-4a2b-93a2-729f684c4afa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:20.079347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1176/2337] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:39:20 2022] POST /identity/v3/auth/tokens => generated 3372 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:20.090334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4c892c-5f1e-4df8-a6ce-86e364bdbc9d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:20.101512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af4c892c-5f1e-4df8-a6ce-86e364bdbc9d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:20.105511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:20.110807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:20.110807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:20.110807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:20.116703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:20.118690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:20.129363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1162/2338] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:39:20 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:39:30.062754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=68 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:30.125306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:30.208823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ff7687-67f5-4ee7-a23d-c4a439879f84 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:30.209107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ff7687-67f5-4ee7-a23d-c4a439879f84 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:30.209380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90ff7687-67f5-4ee7-a23d-c4a439879f84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:30.228899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-90ff7687-67f5-4ee7-a23d-c4a439879f84 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:30.230616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-90ff7687-67f5-4ee7-a23d-c4a439879f84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:30.292247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1177/2339] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:39:30 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:30.318286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:30.394475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a6963c-e0f6-4ae8-8508-f07505428876 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:30.399501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:30.406695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:30.406695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:30.406695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:30.409518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:30.411130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:30.420326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1163/2340] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:39:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:39:34.461705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:34.462217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:34.462217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:34.487430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:34.488716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:34.497718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1178/2341] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:39:34 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:34.512155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:34.523111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2776780-864c-4fd1-8926-5658fe117631 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:34.532234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43184235-59d4-4f86-b27d-f329a3c0359e service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:34.532965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43184235-59d4-4f86-b27d-f329a3c0359e service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:34.533242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43184235-59d4-4f86-b27d-f329a3c0359e service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:34.533514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43184235-59d4-4f86-b27d-f329a3c0359e service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:34.543658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43184235-59d4-4f86-b27d-f329a3c0359e service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:34.545692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43184235-59d4-4f86-b27d-f329a3c0359e service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:34.560680 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1164/2342] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:39:34 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:39:44.480585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=28 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:44.554875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=38 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:39:45.739956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:45.756937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc51dee-0e3a-468c-9843-163ce0eaa46b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:45.761262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:45.761948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:45.762243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:45.762513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:45.764446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:45.877418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:45.878889 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:45.890117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1179/2343] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:39:45 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 151 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:39:47.633902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699d856-6d43-4b51-85c6-7745953947eb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:47.634895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699d856-6d43-4b51-85c6-7745953947eb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:47.635347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e699d856-6d43-4b51-85c6-7745953947eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:47.659123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e699d856-6d43-4b51-85c6-7745953947eb None None] MFA Rules not processed for user `421aaa238a3c495096dfc55aba76267c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:47.660675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e699d856-6d43-4b51-85c6-7745953947eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:47.685984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1165/2344] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:39:47 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:48.858221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:48.859600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:48.983576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f73db273-9aca-4032-beba-198f7a9ea239 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:48.989230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:48.989845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:48.990111 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:48.990556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:48.996490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:49.017915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:49.019817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bea764f3-4170-4a42-b335-61f69e671f16 service placement] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:49.033714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1180/2345] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:39:48 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 176 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:39:54.774719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:54.775300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:54.775300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:54.796156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] MFA Rules not processed for user `2f9e55a3ff7c424dbd7a58f0eb92b8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:54.797406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:54.855782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1166/2346] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:39:54 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:55.601007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c108b5-dc9c-44f3-a8d4-59917d1e91c2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:55.601607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c108b5-dc9c-44f3-a8d4-59917d1e91c2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:55.601607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01c108b5-dc9c-44f3-a8d4-59917d1e91c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:55.628963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-01c108b5-dc9c-44f3-a8d4-59917d1e91c2 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:39:55.630178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-01c108b5-dc9c-44f3-a8d4-59917d1e91c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:39:55.637860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1181/2347] 10.222.0.64 () {58 vars in 967 bytes} [Tue Sep 27 16:39:55 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:39:55.651978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:39:55.688777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-680d528c-0130-4a1c-8b4b-0a0d9da0fc2a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:39:55.688777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:39:55.688777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:39:55.688777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:39:55.689346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:39:55.689346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:39:55.690747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:39:55.723278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1167/2348] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:39:55 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:40:00.403331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2014d690-69b3-41e4-b206-3eb8ab0f37b9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:00.403869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2014d690-69b3-41e4-b206-3eb8ab0f37b9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:00.403869 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2014d690-69b3-41e4-b206-3eb8ab0f37b9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:00.431829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-2014d690-69b3-41e4-b206-3eb8ab0f37b9 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:40:00.431829 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2014d690-69b3-41e4-b206-3eb8ab0f37b9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:00.441579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1182/2349] 10.222.0.64 () {58 vars in 967 bytes} [Tue Sep 27 16:40:00 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:40:00.453955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:40:00.464570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36048193-070d-4df1-b18d-32b3f4e37e9b service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:40:00.470190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a45ad02-2a2e-4ec0-811a-efa636171879 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:40:00.470799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a45ad02-2a2e-4ec0-811a-efa636171879 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:00.471090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a45ad02-2a2e-4ec0-811a-efa636171879 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:00.471360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a45ad02-2a2e-4ec0-811a-efa636171879 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:00.480050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a45ad02-2a2e-4ec0-811a-efa636171879 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:40:00.482430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a45ad02-2a2e-4ec0-811a-efa636171879 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:40:00.490996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1168/2350] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:40:00 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:40:02.139378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab71bfd-04e8-4e5b-8221-62e3118ee644 None None] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:02.139948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab71bfd-04e8-4e5b-8221-62e3118ee644 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:02.139948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ab71bfd-04e8-4e5b-8221-62e3118ee644 None None] PATH_INFO: `/` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:02.141297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1183/2351] 10.222.0.64 () {54 vars in 921 bytes} [Tue Sep 27 16:40:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Sep 27 16:40:02.169434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:02.170240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:02.170600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:02.193887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] MFA Rules not processed for user `4a39bfa348af4f499621fcb5d612e82c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:40:02.193887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:02.202655 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1169/2352] 10.222.0.64 () {58 vars in 1018 bytes} [Tue Sep 27 16:40:02 2022] POST /identity/v3/auth/tokens => generated 3308 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:40:02.214313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab71bfd-04e8-4e5b-8221-62e3118ee644 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:40:02.232436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ab71bfd-04e8-4e5b-8221-62e3118ee644 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:40:02.237447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:40:02.238037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:02.238314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:02.238609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:02.247848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:40:02.249455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:40:02.259140 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1184/2353] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:40:02 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:40:02.292160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:40:02.294033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:02.361835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d94d23-bdd1-4617-aaa8-5b4e2787821e None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:40:02.366363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:40:02.367039 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:02.367321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:02.367597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] PATH_INFO: `/v3/users/f7e32dbc314d4953b3e3c21cbb481367` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:02.368724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f7e32dbc314d4953b3e3c21cbb481367)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:40:02.370639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:40:02.554333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f7e32dbc314d4953b3e3c21cbb481367'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:02.563455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f7e32dbc314d4953b3e3c21cbb481367'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:02.564180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f7e32dbc314d4953b3e3c21cbb481367'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:02.564874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f7e32dbc314d4953b3e3c21cbb481367'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:02.631670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f7e32dbc314d4953b3e3c21cbb481367'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:02.637495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:02.651341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1170/2354] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:40:02 2022] DELETE /identity/v3/users/f7e32dbc314d4953b3e3c21cbb481367 => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:40:02.907855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:40:02.920704 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-762b1b8b-21f8-46ee-aada-16e17c530088 service placement] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:40:02.980471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:02.986972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:40:02.987683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:02.988109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:02.988391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] PATH_INFO: `/v3/projects/54a4b85dba3b48b68e25c190b98b6197` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:02.989525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] RBAC: Authorizing `identity:delete_project(project_id=54a4b85dba3b48b68e25c190b98b6197)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:40:02.991476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:40:03.100004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54a4b85dba3b48b68e25c190b98b6197', 'request_id': 'req-8c7ef03c-4487-4144-8e49-53e9bbe5c739'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:03.196631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54a4b85dba3b48b68e25c190b98b6197', 'request_id': 'req-8c7ef03c-4487-4144-8e49-53e9bbe5c739'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:03.204042 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54a4b85dba3b48b68e25c190b98b6197', 'request_id': 'req-8c7ef03c-4487-4144-8e49-53e9bbe5c739'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:03.214213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] The token cache is being invalidate because project 54a4b85dba3b48b68e25c190b98b6197 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:40:03.214504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:03.215966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-8c7ef03c-4487-4144-8e49-53e9bbe5c739 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:40:03.217524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1185/2355] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:40:02 2022] DELETE /identity/v3/projects/54a4b85dba3b48b68e25c190b98b6197 => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:40:12.784630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=2,keystone:SELECT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:13.211152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=25,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:24.274433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:40:24.276067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:24.373951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a43dfcc8-8c2b-4d82-bf9a-ca58641aced3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:40:24.457962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:40:24.460686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:24.460686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:24.460686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:24.464186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:24.561546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:40:24.564244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ca904ab-e936-4b39-bd61-4f3c16d8a2a0 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:40:24.578436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1171/2356] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:40:24 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:40:29.678239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0256bc61-5e04-4e83-bcd8-52b6b9c62db4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:29.681089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0256bc61-5e04-4e83-bcd8-52b6b9c62db4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:29.681089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0256bc61-5e04-4e83-bcd8-52b6b9c62db4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:29.699105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0256bc61-5e04-4e83-bcd8-52b6b9c62db4 None None] MFA Rules not processed for user `e8d3af2b82b641aea11ebf79677a3af1`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:40:29.700367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0256bc61-5e04-4e83-bcd8-52b6b9c62db4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:29.756544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1186/2357] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:40:29 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:40:34.570886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=70 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:35.247448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:35.247834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:35.248118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:35.270900 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] MFA Rules not processed for user `211efc6d124344c0b6ba83c378425df0`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:40:35.272088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:35.352834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1172/2358] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:40:35 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:40:39.749391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=25 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:45.349716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=25 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:40:58.391202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:40:58.391791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:40:58.391791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:40:58.410961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] MFA Rules not processed for user `1dbfb1a3e99c418098c2b02aea93b923`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:40:58.412093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:40:58.466628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1187/2359] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:40:58 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:41:08.459873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=25 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:41:25.257115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:41:25.258573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:25.270578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9074ac1-be89-4529-9f2c-b03235701f4d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:41:25.274141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:41:25.274743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:25.275048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:25.280464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:25.280464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:25.350150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:41:25.351634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:41:25.363598 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1173/2360] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:41:25 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:41:29.109036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:41:29.119443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7018746a-433a-4aac-bb9f-d83d5a4ce3d8 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:41:29.122957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:41:29.123650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:29.123929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:29.124196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:29.126003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:29.140698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:41:29.142150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cb0cc0-b035-4927-b524-a2672a993128 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:41:29.153183 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1188/2361] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:41:29 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:41:35.357785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=34 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:41:39.148344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=6 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:41:40.635240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:41:40.636659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:40.694349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c713505c-cfeb-4d31-9c9a-cbac3a5e55da service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:41:40.698111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:41:40.698704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:40.699003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:40.699273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:40.701314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:40.713118 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:41:40.714528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:41:40.723767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1174/2362] 10.222.0.64 () {60 vars in 1396 bytes} [Tue Sep 27 16:41:40 2022] GET /identity/v3/auth/tokens?allow_expired=1 => generated 3260 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:41:43.387102 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:43.387558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:43.387727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:43.407897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] MFA Rules not processed for user `2f9e55a3ff7c424dbd7a58f0eb92b8ce`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:41:43.409083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:43.473244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1189/2363] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:41:43 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:41:48.991854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:41:48.993321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:49.006073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab6720b-8f3d-456e-82c5-4bc44c5445fd service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:41:49.009649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:41:49.010247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:49.010610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:49.010866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:49.012554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:49.024492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:41:49.027179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:41:49.035856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1175/2364] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:41:48 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 45 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:41:52.956120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:41:52.957218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:53.022008 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d277b20b-132d-405e-9257-8c82d3e18277 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:41:53.026578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:41:53.030433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:53.030433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:53.030433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] PATH_INFO: `/v3/users/211efc6d124344c0b6ba83c378425df0` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:53.030433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] RBAC: Authorizing `identity:delete_user(user_id=211efc6d124344c0b6ba83c378425df0)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:41:53.031128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:41:53.226426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '211efc6d124344c0b6ba83c378425df0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.227201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '211efc6d124344c0b6ba83c378425df0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.234324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '211efc6d124344c0b6ba83c378425df0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.235021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '211efc6d124344c0b6ba83c378425df0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.314896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c95329e1-ba9f-4d72-a306-d25263fca142 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '211efc6d124344c0b6ba83c378425df0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.336327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1190/2365] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:41:52 2022] DELETE /identity/v3/users/211efc6d124344c0b6ba83c378425df0 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:41:53.578423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:41:53.595360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3302c601-2c1f-422b-b8fe-593d6a97f977 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:41:53.658122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:41:53.658746 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:53.659051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:53.659319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] PATH_INFO: `/v3/projects/a06947e0733946bb8e266382c0d4d0ce` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:53.660419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] RBAC: Authorizing `identity:delete_project(project_id=a06947e0733946bb8e266382c0d4d0ce)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:41:53.663044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:41:53.818593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a06947e0733946bb8e266382c0d4d0ce', 'request_id': 'req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.824427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a06947e0733946bb8e266382c0d4d0ce', 'request_id': 'req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.897995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a06947e0733946bb8e266382c0d4d0ce', 'request_id': 'req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.910203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] The token cache is being invalidate because project a06947e0733946bb8e266382c0d4d0ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:41:53.910494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.911916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:41:53.913857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1176/2366] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:41:53 2022] DELETE /identity/v3/projects/a06947e0733946bb8e266382c0d4d0ce => generated 0 bytes in 336 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:41:59.738898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:41:59.739723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:41:59.740043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:41:59.759816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] MFA Rules not processed for user `e8d3af2b82b641aea11ebf79677a3af1`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:41:59.761036 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:41:59.780125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:41:59.818611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1191/2367] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:41:59 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:00.121626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:00.123113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:00.145995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:00.180936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f09f315-0fab-484e-bf7a-cb2cf77ef0eb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:00.184490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:00.185088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:00.185363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:00.185630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:00.187352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:00.197352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:00.198766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:00.206615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1177/2368] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:00 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:00.598114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:00.599817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:00.671362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb27eda2-65f5-4e85-9556-802730409927 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:00.676388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:00.677563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:00.677804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:00.678072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:00.680484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:00.691764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:00.694186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:00.704070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1192/2369] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:00 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:01.376004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:01.378303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:01.494478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f8c3782-a28e-457b-a36f-6b7c9f1ad983 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:01.501517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:01.502235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:01.502742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:01.503139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:01.505782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:01.521186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:01.524022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:01.535354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1178/2370] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:01 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 160 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:05.194442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:05.195943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:05.220248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d81f80d8-0ade-4513-bcf3-fcc5c7bcd6b1 service placement] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:05.226561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:05.227201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:05.227520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:05.227855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:05.229438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:05.243383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:05.245000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:05.255897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1193/2371] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:05 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:05.628305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:05.641237 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e42d34f-dcb7-4cee-87c1-fcf1a425d561 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:05.647239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:05.647863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:05.648145 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:05.648417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:05.650557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:05.711193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:05.712808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73183811-0ded-4d60-bf75-093805c58154 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:05.721858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1179/2372] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:05 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:09.035203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:09.046946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5307e65-8daf-4091-9f9f-b38d978ad9be service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:09.052977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:09.053375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:09.053650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:09.053920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:09.055632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:09.068071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:09.069641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:09.080861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1194/2373] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:09 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:12.069734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:12.070264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:12.070395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:12.091271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:12.092472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:12.138370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:12.161115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1180/2374] 10.222.0.64 () {58 vars in 967 bytes} [Tue Sep 27 16:42:12 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:12.332229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:12.338942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:12.410444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d242ab21-d332-49c7-b892-c286d9c67ca5 service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:12.415921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:12.416531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:12.416800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:12.417258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:12.428463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:12.431361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:12.439544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1195/2375] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:42:12 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 27 16:42:21.069439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:21.070932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:21.134436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b9469d-6741-46d8-895b-0077e3f10d50 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:21.139061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:21.139692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:21.139967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:21.140239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] PATH_INFO: `/v3/users/978e8037ce1d4742abe66ea12d660479` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:21.141378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] RBAC: Authorizing `identity:delete_user(user_id=978e8037ce1d4742abe66ea12d660479)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:21.143331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:21.338871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '978e8037ce1d4742abe66ea12d660479'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.346290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '978e8037ce1d4742abe66ea12d660479'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.346999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '978e8037ce1d4742abe66ea12d660479'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.347619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '978e8037ce1d4742abe66ea12d660479'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.404169 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '978e8037ce1d4742abe66ea12d660479'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.422478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1181/2376] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:42:21 2022] DELETE /identity/v3/users/978e8037ce1d4742abe66ea12d660479 => generated 0 bytes in 354 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:42:21.654981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:21.666733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5df4f47-5dd9-4283-8c5a-7bfa4e9d87de service swift] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:21.689939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:21.732189 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:21.732822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:21.733099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:21.733372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] PATH_INFO: `/v3/projects/d5fb5144fbf64abe8ec6b37c7f0444b3` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:21.734502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] RBAC: Authorizing `identity:delete_project(project_id=d5fb5144fbf64abe8ec6b37c7f0444b3)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:21.736509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:21.852277 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd5fb5144fbf64abe8ec6b37c7f0444b3', 'request_id': 'req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.914117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd5fb5144fbf64abe8ec6b37c7f0444b3', 'request_id': 'req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.921202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd5fb5144fbf64abe8ec6b37c7f0444b3', 'request_id': 'req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.948985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] The token cache is being invalidate because project d5fb5144fbf64abe8ec6b37c7f0444b3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:42:21.949094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.949760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ae959517-cd0f-4d78-ac62-4bfe1a66aa1c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:21.951629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1196/2377] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:42:21 2022] DELETE /identity/v3/projects/d5fb5144fbf64abe8ec6b37c7f0444b3 => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:42:21.955343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:21.956851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:21.971367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-013367ea-4af5-4841-8421-0b8606c2d264 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:22.029412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:22.030044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:22.030324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:22.030923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] PATH_INFO: `/v3/users/883ebd32de2e4cd9a21df28b7297e100` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:22.032178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=883ebd32de2e4cd9a21df28b7297e100)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:22.034219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:22.049380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:22.244611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '883ebd32de2e4cd9a21df28b7297e100'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:22.251642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '883ebd32de2e4cd9a21df28b7297e100'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:22.252541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '883ebd32de2e4cd9a21df28b7297e100'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:22.253253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '883ebd32de2e4cd9a21df28b7297e100'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:22.346462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '883ebd32de2e4cd9a21df28b7297e100'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:22.366703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1182/2378] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:42:21 2022] DELETE /identity/v3/users/883ebd32de2e4cd9a21df28b7297e100 => generated 0 bytes in 412 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:42:22.589609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:22.592015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:22.592015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:22.610376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:22.611685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:22.646661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:22.648178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:22.686348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1197/2379] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:42:22 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:22.698917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:22.700901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:22.737834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d9b2c64-4c09-4bee-9e3d-dd2456912e31 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:22.743293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:22.743921 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:22.744200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:22.744472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] PATH_INFO: `/v3/projects/adcfb47edc514af082ac07df9bfc3dbc` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:22.745629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] RBAC: Authorizing `identity:delete_project(project_id=adcfb47edc514af082ac07df9bfc3dbc)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:22.747563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:22.763083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:22.777581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7499113-d417-44fa-bd70-97efcf43906c None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:22.782761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf79a17-0c5a-4960-b88c-9bc1e37dc6d9 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:22.783382 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf79a17-0c5a-4960-b88c-9bc1e37dc6d9 service swift] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:22.783656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf79a17-0c5a-4960-b88c-9bc1e37dc6d9 service swift] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:22.783919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf79a17-0c5a-4960-b88c-9bc1e37dc6d9 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:22.793370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf79a17-0c5a-4960-b88c-9bc1e37dc6d9 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:22.794916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-caf79a17-0c5a-4960-b88c-9bc1e37dc6d9 service swift] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:22.804425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1198/2380] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:42:22 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Sep 27 16:42:22.915342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adcfb47edc514af082ac07df9bfc3dbc', 'request_id': 'req-861d1bca-7092-43d9-b22a-3fdb743c652a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:22.921092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adcfb47edc514af082ac07df9bfc3dbc', 'request_id': 'req-861d1bca-7092-43d9-b22a-3fdb743c652a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:23.027346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adcfb47edc514af082ac07df9bfc3dbc', 'request_id': 'req-861d1bca-7092-43d9-b22a-3fdb743c652a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:23.047590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] The token cache is being invalidate because project adcfb47edc514af082ac07df9bfc3dbc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:42:23.048069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:23.062319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-861d1bca-7092-43d9-b22a-3fdb743c652a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:23.066416 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1183/2381] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:42:22 2022] DELETE /identity/v3/projects/adcfb47edc514af082ac07df9bfc3dbc => generated 0 bytes in 420 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:42:23.072185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.072513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.072785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.105230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:23.106524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:23.209822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1199/2382] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:23.217261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.217811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.218256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.242031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:23.243649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:23.275372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1184/2383] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 60 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:23.280255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.296213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b959589-d9be-4d2a-9275-96295c3645f2 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:23.304793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:23.305412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.305697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.305966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.306626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:42:23.306966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:23.308601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:23.321520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1200/2384] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:42:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:42:23.329253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.337119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9d6205c-4bd1-45d0-96b9-158df9b8a7b5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:23.347871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:23.348809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.349080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.349372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.350057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:23.352029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:23.365282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:23.455685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1185/2385] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:42:23 2022] POST /identity/v3/projects => generated 371 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:42:23.460509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.474375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab61f6e9-19ed-424d-8c85-8248b936c341 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:23.545572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:23.546192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.549314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.549314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.549314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:23.549728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:23.623808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1201/2386] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:42:23 2022] POST /identity/v3/users => generated 286 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:42:23.628108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.643783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88085936-b5b8-47ab-8b0a-3eb689d86636 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:23.649072 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:23.649897 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.650330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.650613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.652384 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:23.653030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:23.669245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.669689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1186/2387] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:42:23 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:42:23.680947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d49a7b-2379-4bc6-91de-52f01e47a8d9 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:23.685771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1937e286-2be1-4c04-94a0-99d4bffeb508 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:23.686368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1937e286-2be1-4c04-94a0-99d4bffeb508 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.686938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1937e286-2be1-4c04-94a0-99d4bffeb508 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.686938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1937e286-2be1-4c04-94a0-99d4bffeb508 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.687570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1937e286-2be1-4c04-94a0-99d4bffeb508 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:23.689764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1937e286-2be1-4c04-94a0-99d4bffeb508 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:23.707580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1202/2388] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:42:23 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:42:23.709233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.720966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82d84d0d-e997-4cd8-b0fb-a5316f24e3c8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:23.728025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:23.728696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.728978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.729267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] PATH_INFO: `/v3/projects/eb77c9fe25cb4abda83cb3d2e665557a/users/af5c944405e8435f9e0eccaaee764cd2/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.748780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] RBAC: Authorizing `identity:create_grant(project_id=eb77c9fe25cb4abda83cb3d2e665557a, user_id=af5c944405e8435f9e0eccaaee764cd2, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:23.751565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:23.858570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1187/2389] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:42:23 2022] PUT /identity/v3/projects/eb77c9fe25cb4abda83cb3d2e665557a/users/af5c944405e8435f9e0eccaaee764cd2/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:42:23.870281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520d2cb0-6764-40f7-9208-79fb868fad0b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:23.870281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520d2cb0-6764-40f7-9208-79fb868fad0b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:23.870741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-520d2cb0-6764-40f7-9208-79fb868fad0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:23.898356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-520d2cb0-6764-40f7-9208-79fb868fad0b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:23.899668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-520d2cb0-6764-40f7-9208-79fb868fad0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:23.953614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:23.966991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1203/2390] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:23.979336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:23.984241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:24.075105 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-546bb791-7796-4a49-9816-d754d3a63455 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:24.080859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:24.081673 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:24.082527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:24.084007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:24.097094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:24.099019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:24.110911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1188/2391] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:23 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:31.494822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:31.495779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:31.496280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:31.596173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] MFA Rules not processed for user `af5c944405e8435f9e0eccaaee764cd2`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:31.598165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:31.764354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1204/2392] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:31 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:31.770407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:31.925347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91217943-1caf-4721-b523-509c20ba82fa service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:31.925347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:31.925347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:31.925347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:31.926889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:31.926889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:31.926889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:32.067672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1189/2393] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:42:31 2022] POST /identity/v3/projects => generated 371 bytes in 298 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:42:32.071773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:32.084139 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87f4285b-856a-41ee-8822-ea8015d96796 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:32.158496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:32.159459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:32.159459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:32.159655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:32.160701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:32.162721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:32.293552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1205/2394] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:42:32 2022] POST /identity/v3/users => generated 292 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:42:32.297675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:32.313086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ace2c5dd-a1aa-46b4-8ea6-573609133eb5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:32.352510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:32.353983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:32.354495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:32.355146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:32.356256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:32.358162 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:32.399578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:32.408740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1190/2395] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:42:32 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:42:32.415094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-966555de-6f83-4e05-9646-02aaa99337ee admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:32.434492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:32.436565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:32.436856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:32.440322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] PATH_INFO: `/v3/projects/7136a363bf5b4311ab7fa5a35013e5aa/users/88bed0eae4494d248708a57d065287ac/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:32.490249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] RBAC: Authorizing `identity:create_grant(project_id=7136a363bf5b4311ab7fa5a35013e5aa, user_id=88bed0eae4494d248708a57d065287ac, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:32.493031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:32.627104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1206/2396] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:42:32 2022] PUT /identity/v3/projects/7136a363bf5b4311ab7fa5a35013e5aa/users/88bed0eae4494d248708a57d065287ac/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:42:32.632133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:32.657515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89cd822-5859-40a3-8e1c-2ffd002bc245 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:32.766505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74766e80-55c2-429c-a505-b379d5361861 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:32.767102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74766e80-55c2-429c-a505-b379d5361861 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:32.767365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74766e80-55c2-429c-a505-b379d5361861 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:32.767637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74766e80-55c2-429c-a505-b379d5361861 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:32.768284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74766e80-55c2-429c-a505-b379d5361861 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:32.769917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74766e80-55c2-429c-a505-b379d5361861 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:32.785233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1191/2397] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:42:32 2022] GET /identity/v3/roles => generated 1420 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:42:32.796110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:32.809949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52c56d9f-3cb4-4cb5-b9ac-2dabf04854ab admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:32.815660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:32.817776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:32.818177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:32.818818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] PATH_INFO: `/v3/projects/7136a363bf5b4311ab7fa5a35013e5aa/users/88bed0eae4494d248708a57d065287ac/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:32.820940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=7136a363bf5b4311ab7fa5a35013e5aa, user_id=88bed0eae4494d248708a57d065287ac, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:32.824317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:32.849547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1207/2398] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:42:32 2022] PUT /identity/v3/projects/7136a363bf5b4311ab7fa5a35013e5aa/users/88bed0eae4494d248708a57d065287ac/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:42:41.666424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fc80f1-34d6-43d4-9484-00d161885188 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:41.666928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fc80f1-34d6-43d4-9484-00d161885188 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:41.666928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6fc80f1-34d6-43d4-9484-00d161885188 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:41.708857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a6fc80f1-34d6-43d4-9484-00d161885188 None None] MFA Rules not processed for user `88bed0eae4494d248708a57d065287ac`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:41.710516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a6fc80f1-34d6-43d4-9484-00d161885188 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:41.736377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:41.821519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1192/2399] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:41 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:41.836785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:41.838815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:41.885712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:41.926721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0044707e-ac39-4eb1-9d28-ea8617c9ccb3 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:41.932544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae005639-f1ad-47da-a62a-73bf05f263ac service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:41.933162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae005639-f1ad-47da-a62a-73bf05f263ac service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:41.933439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae005639-f1ad-47da-a62a-73bf05f263ac service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:41.933708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae005639-f1ad-47da-a62a-73bf05f263ac service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:41.995780 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae005639-f1ad-47da-a62a-73bf05f263ac service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:41.997385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae005639-f1ad-47da-a62a-73bf05f263ac service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:42.007129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1208/2400] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:42.615173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:42.615621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:42.616051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:42.642976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] MFA Rules not processed for user `421aaa238a3c495096dfc55aba76267c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:42.645257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:42.716219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1193/2401] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:42 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:45.329673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32493fe6-de0f-424c-ae14-32fa1a2f4b49 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:45.333736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32493fe6-de0f-424c-ae14-32fa1a2f4b49 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:45.334311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32493fe6-de0f-424c-ae14-32fa1a2f4b49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:45.365791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-32493fe6-de0f-424c-ae14-32fa1a2f4b49 None None] MFA Rules not processed for user `af5c944405e8435f9e0eccaaee764cd2`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:45.367081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-32493fe6-de0f-424c-ae14-32fa1a2f4b49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:45.386110 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1209/2402] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:42:45 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:46.786951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:46.803085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a76e4daa-9953-4611-ad97-70b179e0bd7d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:46.809154 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:46.810183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:46.810478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:46.810740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:46.812589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:46.891161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:46.892443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:46.903228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1194/2403] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:46 2022] GET /identity/v3/auth/tokens => generated 3266 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:52.748743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:52.749766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:52.751294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:52.795575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:42:52.796986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:52.858850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1210/2404] 10.222.0.64 () {58 vars in 1009 bytes} [Tue Sep 27 16:42:52 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:42:52.878235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:52.879646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:52.937984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:52.998668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc4e7b7-25ef-4e89-a178-0f604cbfbae4 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:53.005485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:53.006159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:53.006443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:53.006716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:53.018860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:53.020872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:53.030859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1195/2405] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:42:52 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 153 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:42:53.393137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:53.495491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6539630-effb-4e83-91f9-429becd4c79e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:53.499358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:53.500005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:53.500285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:53.500622 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:53.508870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:53.532499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:42:53.674070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:53.676703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:53.717777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1211/2406] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 325 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:59.130122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:59.131917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:59.262076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1598309-34b0-4341-a80e-f535fb7f8237 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:59.268168 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:59.268933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:59.269962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:59.270307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:59.272248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:59.284883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:59.286661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:59.298240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1196/2407] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:42:59 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:42:59.309977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:59.312354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:42:59.326177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf572b27-e924-4ea0-b65e-742ce708375f service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:42:59.335695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:42:59.343558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:42:59.343558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:42:59.343558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] PATH_INFO: `/v3/users/2f9e55a3ff7c424dbd7a58f0eb92b8ce` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:42:59.344034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] RBAC: Authorizing `identity:delete_user(user_id=2f9e55a3ff7c424dbd7a58f0eb92b8ce)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:42:59.348758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:42:59.546372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '2f9e55a3ff7c424dbd7a58f0eb92b8ce'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:59.548035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '2f9e55a3ff7c424dbd7a58f0eb92b8ce'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:59.557672 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '2f9e55a3ff7c424dbd7a58f0eb92b8ce'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:59.558550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '2f9e55a3ff7c424dbd7a58f0eb92b8ce'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:59.638244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60db1021-92ca-4e7e-9f7d-1ce31e00f874 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '2f9e55a3ff7c424dbd7a58f0eb92b8ce'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:42:59.669962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1212/2408] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:42:59 2022] DELETE /identity/v3/users/2f9e55a3ff7c424dbd7a58f0eb92b8ce => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:42:59.982817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:42:59.996035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c573477d-370b-47ba-b38b-987c248a2f04 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:00.077572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:00.078369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.078651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.078972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] PATH_INFO: `/v3/projects/9288b4e62bc945479636aec307a98746` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.080174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9288b4e62bc945479636aec307a98746)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:00.083439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:00.111963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:00.196932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9288b4e62bc945479636aec307a98746', 'request_id': 'req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:00.203316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9288b4e62bc945479636aec307a98746', 'request_id': 'req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:00.279877 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9288b4e62bc945479636aec307a98746', 'request_id': 'req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:00.293533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] The token cache is being invalidate because project 9288b4e62bc945479636aec307a98746 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:00.306059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:00.306059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ca9ee3ea-8e1d-47e5-a55a-503ec75ee007 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:00.306059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1197/2409] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:42:59 2022] DELETE /identity/v3/projects/9288b4e62bc945479636aec307a98746 => generated 0 bytes in 314 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:00.306339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.306339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.306339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.409726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:00.411353 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:00.434336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:00.526650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1213/2410] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 227 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:00.536599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.536864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.537187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.569224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:00.572254 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:00.586869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1198/2411] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:00.592014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:00.609935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d637d82-4fa5-4ea8-917a-193b9176e834 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:00.615563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:00.616156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.616446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.616705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.617357 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:43:00.617658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:00.619310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:00.640255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1214/2412] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:43:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 49 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:43:00.645618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:00.651492 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4992b30-35b2-4b62-ac69-ffdfbe354239 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:00.656352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:00.656958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.657236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.657498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.658117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:00.660155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:00.766968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1199/2413] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:00 2022] POST /identity/v3/projects => generated 365 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:00.771354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:00.782487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1790110-5007-4b62-b70d-bba601160292 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:00.844508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:00.845252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.845595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.845940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.846647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:00.849650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:00.928266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1215/2414] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:00 2022] POST /identity/v3/users => generated 283 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:00.938049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:00.956982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab2e98c-fd48-456e-9524-febef705f21a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:00.986724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:00.987378 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:00.987659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:00.988523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:00.989622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:00.997542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.010428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1200/2415] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:00 2022] GET /identity/v3/roles => generated 1420 bytes in 78 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:01.015589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.032395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd175071-e580-403d-b989-a7415eff6a1f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.038554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19c26f33-9ca9-4f98-8e4f-a7e38ed93fbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.039228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c26f33-9ca9-4f98-8e4f-a7e38ed93fbb admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.039504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c26f33-9ca9-4f98-8e4f-a7e38ed93fbb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.039770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19c26f33-9ca9-4f98-8e4f-a7e38ed93fbb admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.040410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c26f33-9ca9-4f98-8e4f-a7e38ed93fbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.042978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19c26f33-9ca9-4f98-8e4f-a7e38ed93fbb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.061728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1216/2416] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:01 2022] GET /identity/v3/roles => generated 1420 bytes in 48 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:01.065888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.078495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96c0051-c4f5-48cc-a1aa-8177a4377c2d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.084230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.084899 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.085176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.085448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] PATH_INFO: `/v3/projects/88e066e0b31841738fa7d211fc91149c/users/054643225cf04a9e845eaacadcb91c11/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.103456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] RBAC: Authorizing `identity:create_grant(project_id=88e066e0b31841738fa7d211fc91149c, user_id=054643225cf04a9e845eaacadcb91c11, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.106379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.175040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1201/2417] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:01 2022] PUT /identity/v3/projects/88e066e0b31841738fa7d211fc91149c/users/054643225cf04a9e845eaacadcb91c11/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:01.186299 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.186597 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.186891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.218952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] MFA Rules not processed for user `054643225cf04a9e845eaacadcb91c11`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:01.220251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:01.294570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1217/2418] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:01 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:01.298663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.364870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef412c1f-3b70-4ed2-905f-03f647f68d4b admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.369695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.370397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.370587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.370891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.371537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.373478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.487474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1202/2419] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:01 2022] POST /identity/v3/projects => generated 363 bytes in 189 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:01.491937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.505141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e5ca831-a6b9-4b82-a541-a3e2d0012ab7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.518173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:01.569017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.569017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.569017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.569017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.569501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.570283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.686726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.689398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1218/2420] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:01 2022] POST /identity/v3/users => generated 288 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:01.700385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26a53831-75ac-4e9e-b953-8be4054f27c2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.707902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.708622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.708972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.709312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.710134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.711800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.725823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1203/2421] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:01 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:43:01.735667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.746943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10f1a31f-e369-4489-bbbd-e3a4eb5afbdc admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.752250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.752952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.753235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.753639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] PATH_INFO: `/v3/projects/61738cb4c7cb47fa8dbad27008c10556/users/406449cba17547498e34742f8ca205bd/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.769667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=61738cb4c7cb47fa8dbad27008c10556, user_id=406449cba17547498e34742f8ca205bd, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.772426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.845540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1219/2422] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:01 2022] PUT /identity/v3/projects/61738cb4c7cb47fa8dbad27008c10556/users/406449cba17547498e34742f8ca205bd/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:01.850208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.860192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55c42008-49f2-48db-97bc-6e70bf19b27a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.869094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:01.921792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc44b853-24bd-4f48-9d9f-8a8d2b1f599c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.922412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc44b853-24bd-4f48-9d9f-8a8d2b1f599c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.922689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc44b853-24bd-4f48-9d9f-8a8d2b1f599c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.922986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc44b853-24bd-4f48-9d9f-8a8d2b1f599c admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.923628 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc44b853-24bd-4f48-9d9f-8a8d2b1f599c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.925216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc44b853-24bd-4f48-9d9f-8a8d2b1f599c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.935714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1204/2423] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:01 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:01.940792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:01.952063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9ac960-0a5c-4d7e-b539-66776cbaa4ce admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:01.958013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:01.958719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.959062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.959373 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] PATH_INFO: `/v3/projects/61738cb4c7cb47fa8dbad27008c10556/users/406449cba17547498e34742f8ca205bd/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:01.961348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] RBAC: Authorizing `identity:create_grant(project_id=61738cb4c7cb47fa8dbad27008c10556, user_id=406449cba17547498e34742f8ca205bd, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:01.964133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:01.976589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1220/2424] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:01 2022] PUT /identity/v3/projects/61738cb4c7cb47fa8dbad27008c10556/users/406449cba17547498e34742f8ca205bd/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:01.987071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:01.987356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:01.987633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:02.016128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] MFA Rules not processed for user `406449cba17547498e34742f8ca205bd`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:02.017411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:02.094419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1205/2425] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:01 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:02.107543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:02.109347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:02.194742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad9f5278-c79c-464b-95b7-76dc10544c7f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:02.198711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d346530a-20ec-400e-bc0d-0f23767b495f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:02.199363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d346530a-20ec-400e-bc0d-0f23767b495f service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:02.199641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d346530a-20ec-400e-bc0d-0f23767b495f service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:02.199929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d346530a-20ec-400e-bc0d-0f23767b495f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:02.210229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d346530a-20ec-400e-bc0d-0f23767b495f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:02.211686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d346530a-20ec-400e-bc0d-0f23767b495f service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:02.226997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1221/2426] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:02 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:03.181018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:03.258445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e62c79ac-5bda-412a-bce4-fb2daef505e0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:03.263670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:03.264314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:03.264600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:03.264876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] PATH_INFO: `/v3/users/054643225cf04a9e845eaacadcb91c11` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:03.266029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=054643225cf04a9e845eaacadcb91c11)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:03.268222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:03.470491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:03.494637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '054643225cf04a9e845eaacadcb91c11'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:03.502065 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '054643225cf04a9e845eaacadcb91c11'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:03.502738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '054643225cf04a9e845eaacadcb91c11'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:03.503410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '054643225cf04a9e845eaacadcb91c11'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:03.632031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '054643225cf04a9e845eaacadcb91c11'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:03.652890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1206/2427] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:03 2022] DELETE /identity/v3/users/054643225cf04a9e845eaacadcb91c11 => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:03.656550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:03.656990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:03.657115 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:03.679657 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:03.680808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:03.739165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:03.756049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1222/2428] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:03.765656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:03.767178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:03.849500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72524a47-8434-41c7-b810-dbd07cfefdc8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:03.853706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:03.854314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:03.854594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:03.854904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:03.866828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:03.868332 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:03.881298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1207/2429] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:03 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:05.315303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:05.317820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:05.330131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c66b6c6-e342-411d-9c08-b7bd708fd5c9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:05.335642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:05.336272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:05.336545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:05.336811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] PATH_INFO: `/v3/projects/88e066e0b31841738fa7d211fc91149c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:05.338150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=88e066e0b31841738fa7d211fc91149c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:05.340240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:05.462661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '88e066e0b31841738fa7d211fc91149c', 'request_id': 'req-c50357d9-805b-4327-9ea1-7a784e359b0e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.557812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '88e066e0b31841738fa7d211fc91149c', 'request_id': 'req-c50357d9-805b-4327-9ea1-7a784e359b0e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.566990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '88e066e0b31841738fa7d211fc91149c', 'request_id': 'req-c50357d9-805b-4327-9ea1-7a784e359b0e'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.579444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] The token cache is being invalidate because project 88e066e0b31841738fa7d211fc91149c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:05.579808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.580530 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.583016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1223/2430] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:05 2022] DELETE /identity/v3/projects/88e066e0b31841738fa7d211fc91149c => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:05.586974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:05.597572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2dec2d-2402-4ebf-acb1-21756e306bfa service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:05.657570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:05.658207 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:05.658487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:05.658766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] PATH_INFO: `/v3/users/406449cba17547498e34742f8ca205bd` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:05.659948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] RBAC: Authorizing `identity:delete_user(user_id=406449cba17547498e34742f8ca205bd)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:05.662089 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:05.877291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:05.883934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '406449cba17547498e34742f8ca205bd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.895442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '406449cba17547498e34742f8ca205bd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.896275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '406449cba17547498e34742f8ca205bd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.896953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '406449cba17547498e34742f8ca205bd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.971230 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-49042a06-e78b-420f-a14e-df772311804e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '406449cba17547498e34742f8ca205bd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:05.990194 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1208/2431] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:05 2022] DELETE /identity/v3/users/406449cba17547498e34742f8ca205bd => generated 0 bytes in 404 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:06.479536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:06.481032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:06.494656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50357d9-805b-4327-9ea1-7a784e359b0e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:06.554751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:06.555706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:06.555982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:06.556251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] PATH_INFO: `/v3/projects/61738cb4c7cb47fa8dbad27008c10556` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:06.557498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] RBAC: Authorizing `identity:delete_project(project_id=61738cb4c7cb47fa8dbad27008c10556)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:06.559526 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:06.686712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61738cb4c7cb47fa8dbad27008c10556', 'request_id': 'req-abb95b88-0c2b-4f0d-bd55-734603d08cef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:06.758858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61738cb4c7cb47fa8dbad27008c10556', 'request_id': 'req-abb95b88-0c2b-4f0d-bd55-734603d08cef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:06.766731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61738cb4c7cb47fa8dbad27008c10556', 'request_id': 'req-abb95b88-0c2b-4f0d-bd55-734603d08cef'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:06.779876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] The token cache is being invalidate because project 61738cb4c7cb47fa8dbad27008c10556 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:06.780173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:06.780854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-abb95b88-0c2b-4f0d-bd55-734603d08cef admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:06.782351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1224/2432] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:06 2022] DELETE /identity/v3/projects/61738cb4c7cb47fa8dbad27008c10556 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:06.787847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:06.788342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:06.788451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:06.808710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:06.809912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:06.868954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1209/2433] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:06.878295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:06.878629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:06.879195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:06.888125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:06.900587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:06.901652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:06.913484 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1225/2434] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:06 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:06.915073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:06.925893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-993e09e5-7c21-4af5-875d-8c8558d40a98 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:06.930663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:06.931302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:06.931580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:06.931851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:06.932549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:43:06.932856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:06.934444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:06.946394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1210/2435] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:43:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:43:06.950701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:06.960105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8345fa3-8bfd-4bd6-a878-a844f5467703 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:06.964925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:06.965522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:06.968241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:06.968241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:06.968241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:06.968604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.107171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1226/2436] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:06 2022] POST /identity/v3/projects => generated 377 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:07.108314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.119878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca8dc92-434d-49bb-a556-4e28ad7fe2d0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.198888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.199502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.199782 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.200058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.200691 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.202747 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.314222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1211/2437] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:07 2022] POST /identity/v3/users => generated 289 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:07.318891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.329255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd73f362-d700-442c-a57f-c29e15e3eec1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.342431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.343604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.343604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.343604 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.344278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.346067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.358275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1227/2438] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:07 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:07.362263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.373850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a7f04a7-b90b-471f-8cfd-870fc192d968 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.379121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-820def08-5438-456b-bfc7-a3065b9ec99f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.379732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820def08-5438-456b-bfc7-a3065b9ec99f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.380005 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820def08-5438-456b-bfc7-a3065b9ec99f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.380272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-820def08-5438-456b-bfc7-a3065b9ec99f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.380902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820def08-5438-456b-bfc7-a3065b9ec99f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.382481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-820def08-5438-456b-bfc7-a3065b9ec99f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.393219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1212/2439] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:07.397626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.408024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5788c68-4808-4acc-82f0-b89d07795de2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.412931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.416445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.416445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.416445 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] PATH_INFO: `/v3/projects/0ef72b1313584b3a92e618c6a5db8c93/users/e4a1a4dbfea64605aeb9560118aa7a0d/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.429160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ef72b1313584b3a92e618c6a5db8c93, user_id=e4a1a4dbfea64605aeb9560118aa7a0d, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.431900 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.491423 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1228/2440] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:07 2022] PUT /identity/v3/projects/0ef72b1313584b3a92e618c6a5db8c93/users/e4a1a4dbfea64605aeb9560118aa7a0d/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:07.501713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.502020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.502291 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.530269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] MFA Rules not processed for user `e4a1a4dbfea64605aeb9560118aa7a0d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:07.531584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:07.542290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:07.598640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1213/2441] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:07 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:07.603397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.667346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65c609a7-a38f-454c-b3f5-34a4b999d769 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.672040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.672636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.672924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.673195 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.673819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.676856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.776822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1229/2442] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:07 2022] POST /identity/v3/projects => generated 375 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:07.782728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.790562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf98a82-1cac-4650-a827-974d4d71566b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.849845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.850462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.850743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.851058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.851736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.853705 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.925499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1214/2443] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:07 2022] POST /identity/v3/users => generated 294 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:07.929660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.940082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b784065-2ba7-4e32-b26d-1f86757684df admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.944868 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.945485 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.945760 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.946049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.946681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.948294 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:07.959288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1230/2444] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:07.963429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:07.974373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67cbf721-fc9a-49f7-950e-b027f7535053 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:07.979068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:07.979753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:07.980031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:07.980304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] PATH_INFO: `/v3/projects/6e6a916acbec45469085afadc02a8647/users/fc4eda20cbf14d31b7b50b72e8c40de8/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:07.995106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e6a916acbec45469085afadc02a8647, user_id=fc4eda20cbf14d31b7b50b72e8c40de8, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:07.997995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:08.103958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1215/2445] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:07 2022] PUT /identity/v3/projects/6e6a916acbec45469085afadc02a8647/users/fc4eda20cbf14d31b7b50b72e8c40de8/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:08.108775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:08.118142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85d33118-6f5b-4954-a8e9-44a1973648d1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:08.184267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f4b299e-9e13-4fff-8c79-4a3c75203644 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:08.184901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4b299e-9e13-4fff-8c79-4a3c75203644 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:08.185173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4b299e-9e13-4fff-8c79-4a3c75203644 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:08.185446 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f4b299e-9e13-4fff-8c79-4a3c75203644 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:08.186217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4b299e-9e13-4fff-8c79-4a3c75203644 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:08.189524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f4b299e-9e13-4fff-8c79-4a3c75203644 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:08.203716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1231/2446] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:08 2022] GET /identity/v3/roles => generated 1420 bytes in 93 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:08.205135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:08.216585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-074f9af6-7ae7-4a07-a615-a9e56c96f788 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:08.221750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:08.222442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:08.222726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:08.223040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] PATH_INFO: `/v3/projects/6e6a916acbec45469085afadc02a8647/users/fc4eda20cbf14d31b7b50b72e8c40de8/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:08.225220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e6a916acbec45469085afadc02a8647, user_id=fc4eda20cbf14d31b7b50b72e8c40de8, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:08.227884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:08.238766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1216/2447] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:08 2022] PUT /identity/v3/projects/6e6a916acbec45469085afadc02a8647/users/fc4eda20cbf14d31b7b50b72e8c40de8/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:08.249693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c24ee6-293a-4fd4-8e8b-ff93ea284897 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:08.250062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c24ee6-293a-4fd4-8e8b-ff93ea284897 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:08.250410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4c24ee6-293a-4fd4-8e8b-ff93ea284897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:08.273859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:08.282240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c4c24ee6-293a-4fd4-8e8b-ff93ea284897 None None] MFA Rules not processed for user `fc4eda20cbf14d31b7b50b72e8c40de8`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:08.283548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c4c24ee6-293a-4fd4-8e8b-ff93ea284897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:08.366714 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1232/2448] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:08 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 118 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:08.377519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:08.387012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:08.441800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6caf575-cee0-4f51-9990-6d8c4f2443e4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:08.445836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:08.446442 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:08.446715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:08.447011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:08.464976 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:08.505741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:08.507331 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:08.519225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1217/2449] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:08 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 142 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:09.917650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:09.921881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:09.921881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:09.967593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] MFA Rules not processed for user `b821384a2058464892f0a0f37f34b914`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:09.967593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:10.024684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1233/2450] 10.222.0.64 () {58 vars in 967 bytes} [Tue Sep 27 16:43:09 2022] POST /identity/v3/auth/tokens => generated 3195 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:10.038417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:10.040016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:10.114917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f9d31e3-3212-41bb-aafa-b5f38c298b28 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:10.119953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '85bbaf87748242bf830ca84267692c82', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:10.120577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:10.120853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:10.121122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:10.130187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:10.131825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:10.141029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1218/2451] 10.222.0.64 () {60 vars in 1384 bytes} [Tue Sep 27 16:43:10 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Sep 27 16:43:15.189315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:15.190675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:15.201928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3caf5141-6452-443c-bb24-9d1a830b5fc6 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:15.205415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:15.206125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:15.206271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:15.206520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:15.215310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:15.216717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:15.226524 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1234/2452] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:15 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:20.137715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=51 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:25.222946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=64 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:32.924135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:32.931701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:32.982983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb43df2-1136-44a7-b12c-aa49e967e7c2 service swift] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:32.986799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:32.987407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:32.987684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:32.987952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:32.989700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:33.055266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:33.057062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:33.066997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1219/2453] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:32 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 143 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:33.996988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:33.998613 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:34.011862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f03f2eba-f0e7-46d9-8f13-5dc226a53cfd service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:34.015691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:34.016906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:34.016906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:34.016906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:34.018517 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:34.089557 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:34.091076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:34.101661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1235/2454] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:33 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:37.703282 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:37.704732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:37.719788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-265af2ed-4167-44db-b0e6-53652d8a68c6 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:37.725061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:37.725696 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:37.727163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:37.727163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] PATH_INFO: `/v3/users/e8d3af2b82b641aea11ebf79677a3af1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:37.727422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8d3af2b82b641aea11ebf79677a3af1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:37.729957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:37.933533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8d3af2b82b641aea11ebf79677a3af1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:37.940496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8d3af2b82b641aea11ebf79677a3af1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:37.941112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8d3af2b82b641aea11ebf79677a3af1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:37.941710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8d3af2b82b641aea11ebf79677a3af1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.013350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-3634029f-2689-4e89-8d53-0e214e2a7745 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8d3af2b82b641aea11ebf79677a3af1'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.049947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1220/2455] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:37 2022] DELETE /identity/v3/users/e8d3af2b82b641aea11ebf79677a3af1 => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:38.264216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:38.274951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7331e6c-e943-476a-a5e5-ef5f594341c2 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:38.345352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:38.346052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:38.346338 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:38.346610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] PATH_INFO: `/v3/projects/73fad530905b485cb7141793f6130a9e` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:38.347785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73fad530905b485cb7141793f6130a9e)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:38.350514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:38.469221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73fad530905b485cb7141793f6130a9e', 'request_id': 'req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.559178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73fad530905b485cb7141793f6130a9e', 'request_id': 'req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.566166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73fad530905b485cb7141793f6130a9e', 'request_id': 'req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.577295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] The token cache is being invalidate because project 73fad530905b485cb7141793f6130a9e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:38.577688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.578319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-3cf8e0e3-c875-4880-a87c-178d96f6c7d0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:38.579910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1236/2456] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:38 2022] DELETE /identity/v3/projects/73fad530905b485cb7141793f6130a9e => generated 0 bytes in 316 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:38.585195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:38.585483 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:38.585761 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:38.606015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:38.607304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:38.625878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:38.668427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1221/2457] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:38.675982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:38.676260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:38.676529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:38.696748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:38.697946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:38.706199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1237/2458] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:38 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:38.710495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:38.720013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b3a0692-1d7a-4534-89ed-cc7b51fea6d4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:38.724690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:38.725283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:38.725574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:38.725845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:38.726509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:43:38.726839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:38.731674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:38.740854 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1222/2459] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:43:38 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:43:38.745523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:38.754841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9bc8924-6ad8-45ce-801e-4aa015b87696 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:38.760199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:38.760791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:38.761108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:38.761377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:38.762012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:38.764012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:38.780607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:38.910487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1238/2460] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:38 2022] POST /identity/v3/projects => generated 371 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:38.914725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:38.924963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2719f909-b0ef-44ea-a269-9ddd4e6494b1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:38.981087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:38.981690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:38.981967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:38.982263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:38.982922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:38.984855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:39.079426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1223/2461] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:38 2022] POST /identity/v3/users => generated 286 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:39.083650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:39.094154 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf596a9e-cc40-4e17-8ef4-89fc95d69467 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:39.098730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:39.100169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:39.100451 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:39.100749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:39.101381 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:39.103069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:39.112790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1239/2462] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:39.117312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:39.127177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8daa6082-7ed3-4d79-8002-5a07f352c68a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:39.131955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ea9bf1-5e98-4730-8372-65ab34896038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:39.132570 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea9bf1-5e98-4730-8372-65ab34896038 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:39.132847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea9bf1-5e98-4730-8372-65ab34896038 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:39.133113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea9bf1-5e98-4730-8372-65ab34896038 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:39.133738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ea9bf1-5e98-4730-8372-65ab34896038 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:39.135362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ea9bf1-5e98-4730-8372-65ab34896038 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:39.145821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1224/2463] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:43:39.149703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:39.160062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9525954c-6dfa-4ca6-8a9f-de9af3d91a93 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:39.165656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:39.166231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:39.166533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:39.166827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] PATH_INFO: `/v3/projects/0f08b90b10444c318bcf8c5d4dce2bf6/users/95be235e9a4f41f8ba2df06b38f7e940/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:39.181871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0f08b90b10444c318bcf8c5d4dce2bf6, user_id=95be235e9a4f41f8ba2df06b38f7e940, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:39.184582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:39.285318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1240/2464] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:39 2022] PUT /identity/v3/projects/0f08b90b10444c318bcf8c5d4dce2bf6/users/95be235e9a4f41f8ba2df06b38f7e940/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 136 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:39.296093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:39.296445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:39.296719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:39.323803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] MFA Rules not processed for user `95be235e9a4f41f8ba2df06b38f7e940`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:39.325016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:39.399193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1225/2465] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:39 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:39.410317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:39.412773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:39.485023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c73b7d-0c8a-4cc9-bee4-ac401c88650b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:39.490632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:39.491255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:39.491533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:39.491802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:39.499481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:39.501082 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:39.510571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1241/2466] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:39 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:42.899920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:42.901346 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:42.917920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:42.922730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:42.968602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2840c63b-7f8a-43a8-b4b3-6d24ed9774ee None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:42.974267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:42.974957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:42.975238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:42.976722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] PATH_INFO: `/v3/users/e4a1a4dbfea64605aeb9560118aa7a0d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:42.977882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e4a1a4dbfea64605aeb9560118aa7a0d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:42.980011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:42.985659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aba7e5a-30f1-4bbc-9fb5-22b0158414a9 service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:42.990969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:42.992746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:42.993021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:42.993290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] PATH_INFO: `/v3/users/95be235e9a4f41f8ba2df06b38f7e940` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:42.995033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] RBAC: Authorizing `identity:delete_user(user_id=95be235e9a4f41f8ba2df06b38f7e940)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:42.997762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:43.198914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e4a1a4dbfea64605aeb9560118aa7a0d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.206758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e4a1a4dbfea64605aeb9560118aa7a0d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.207456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e4a1a4dbfea64605aeb9560118aa7a0d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.208060 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e4a1a4dbfea64605aeb9560118aa7a0d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.210684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95be235e9a4f41f8ba2df06b38f7e940'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.211356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95be235e9a4f41f8ba2df06b38f7e940'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.212544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:INSERT=1,keystone:DELETE=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:43.218667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95be235e9a4f41f8ba2df06b38f7e940'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.219315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95be235e9a4f41f8ba2df06b38f7e940'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.281980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-02c606eb-b39d-41d5-bbb8-11789e717e26 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e4a1a4dbfea64605aeb9560118aa7a0d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.300235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1226/2467] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:42 2022] DELETE /identity/v3/users/e4a1a4dbfea64605aeb9560118aa7a0d => generated 0 bytes in 401 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:43.305157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:43.305439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:43.305714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:43.324661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:43.325874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:43.339236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cabf90ed-71f8-41c5-bd2e-e0c3791e7142 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95be235e9a4f41f8ba2df06b38f7e940'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:43.359507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1242/2468] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:42 2022] DELETE /identity/v3/users/95be235e9a4f41f8ba2df06b38f7e940 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:43.363885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:43.364298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:43.364584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:43.384587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:43.385856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:43.393563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1227/2469] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:43.403777 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:43.405113 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:43.449583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:43.451284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1243/2470] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:43 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:43.460733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:43.463527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:43.467496 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b713e4de-8830-478f-8d39-73f097725a95 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:43.517835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:43.518444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:43.518720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:43.519019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:43.520459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a87381cf-b4ea-451c-986c-a03e8e2c135f None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:43.524243 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:43.524848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:43.525125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:43.525396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:43.528306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:43.530031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:43.534888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:43.536303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:43.540548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1228/2471] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:43:43.550306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1244/2472] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:43 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:43.923737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:43.927022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:43.939583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-968337a5-75af-478f-b215-8899207bc8ed service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:43.944861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:43.945530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:43.945837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:43.946106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] PATH_INFO: `/v3/projects/0ef72b1313584b3a92e618c6a5db8c93` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:43.947293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ef72b1313584b3a92e618c6a5db8c93)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:43.949586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:43.957894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:43.959419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:43.974724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e98a3325-ae9e-4f6a-aa70-19e37e90c020 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:43.981540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:43.982201 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:43.982529 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:43.982823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] PATH_INFO: `/v3/projects/0f08b90b10444c318bcf8c5d4dce2bf6` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:43.984003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=0f08b90b10444c318bcf8c5d4dce2bf6)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:43.985949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.077712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ef72b1313584b3a92e618c6a5db8c93', 'request_id': 'req-a00fb79f-785e-4fa7-a46c-90a9632baccd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.083702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ef72b1313584b3a92e618c6a5db8c93', 'request_id': 'req-a00fb79f-785e-4fa7-a46c-90a9632baccd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.106117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0f08b90b10444c318bcf8c5d4dce2bf6', 'request_id': 'req-d723a906-8c35-4233-b3b2-a167aac930bb'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.173441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ef72b1313584b3a92e618c6a5db8c93', 'request_id': 'req-a00fb79f-785e-4fa7-a46c-90a9632baccd'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.186309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] The token cache is being invalidate because project 0ef72b1313584b3a92e618c6a5db8c93 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:44.186597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.187256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.188767 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1229/2473] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:43 2022] DELETE /identity/v3/projects/0ef72b1313584b3a92e618c6a5db8c93 => generated 0 bytes in 266 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:44.192800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.194047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:44.236228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0f08b90b10444c318bcf8c5d4dce2bf6', 'request_id': 'req-d723a906-8c35-4233-b3b2-a167aac930bb'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.244355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0f08b90b10444c318bcf8c5d4dce2bf6', 'request_id': 'req-d723a906-8c35-4233-b3b2-a167aac930bb'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.246598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:INSERT=2,keystone:DELETE=9 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:44.255261 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] The token cache is being invalidate because project 0f08b90b10444c318bcf8c5d4dce2bf6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:44.255548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.256232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d723a906-8c35-4233-b3b2-a167aac930bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.257841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1245/2474] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:43 2022] DELETE /identity/v3/projects/0f08b90b10444c318bcf8c5d4dce2bf6 => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:44.259210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a00fb79f-785e-4fa7-a46c-90a9632baccd admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.262582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d788db-944a-437c-891f-a1c511c128ef None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.262981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d788db-944a-437c-891f-a1c511c128ef None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.263247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07d788db-944a-437c-891f-a1c511c128ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.283178 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-07d788db-944a-437c-891f-a1c511c128ef None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:44.283828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:44.284488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-07d788db-944a-437c-891f-a1c511c128ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:44.316187 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.316865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.317138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.317432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] PATH_INFO: `/v3/users/fc4eda20cbf14d31b7b50b72e8c40de8` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.318554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=fc4eda20cbf14d31b7b50b72e8c40de8)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.320536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.347021 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1246/2475] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:44.356701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.356987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.357259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.377426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:44.378630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:44.387807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1247/2476] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:44 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:44.391911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.401167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-489558c7-1e3c-4a79-b271-db7c7c3be3c1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.406070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.406646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.406955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.407230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.407864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:43:44.408173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.409761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.421713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1248/2477] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:43:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:43:44.426052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.436302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05959e25-b1a1-4cf6-abc0-1f3acab84f6e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.440926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.441487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.441762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.442030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.442639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.444718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.524529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'fc4eda20cbf14d31b7b50b72e8c40de8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.531637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'fc4eda20cbf14d31b7b50b72e8c40de8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.532214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'fc4eda20cbf14d31b7b50b72e8c40de8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.532810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'fc4eda20cbf14d31b7b50b72e8c40de8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.540788 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1249/2478] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:44 2022] POST /identity/v3/projects => generated 373 bytes in 115 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:44.545175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.546807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:44.602548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'fc4eda20cbf14d31b7b50b72e8c40de8'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:44.610745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbc0782-87ba-4a50-9fa1-128047512f43 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.615536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.616162 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.616456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.616748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.617399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.619474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.621023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1230/2479] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:44 2022] DELETE /identity/v3/users/fc4eda20cbf14d31b7b50b72e8c40de8 => generated 0 bytes in 428 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:44.702720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1250/2480] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:44 2022] POST /identity/v3/users => generated 287 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:44.704588 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.768768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcbc9740-0145-4dd6-9727-10ca9d539f05 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.773497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.774088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.774478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.774751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.775354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.776947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.790249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1231/2481] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:44 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:43:44.791707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.801826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12901a61-10cb-47e4-b60f-47a654c2cc3c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.807863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d832940-03af-4f65-bf6b-dccb338d2103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.808462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d832940-03af-4f65-bf6b-dccb338d2103 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.808742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d832940-03af-4f65-bf6b-dccb338d2103 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.809013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d832940-03af-4f65-bf6b-dccb338d2103 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.809651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d832940-03af-4f65-bf6b-dccb338d2103 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.811324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d832940-03af-4f65-bf6b-dccb338d2103 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.826374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1251/2482] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:44 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:44.828125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:44.839424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8283307-f5cd-44f8-ac0a-41cd0d847594 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:44.844032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:44.844670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.844951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.847977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] PATH_INFO: `/v3/projects/de0c370521644f77b1abcc3c0acdb7ce/users/c8b735eb05fa417880f159c384b0b4d7/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.861096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] RBAC: Authorizing `identity:create_grant(project_id=de0c370521644f77b1abcc3c0acdb7ce, user_id=c8b735eb05fa417880f159c384b0b4d7, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:44.863814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:44.932021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1232/2483] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:44 2022] PUT /identity/v3/projects/de0c370521644f77b1abcc3c0acdb7ce/users/c8b735eb05fa417880f159c384b0b4d7/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:44.942507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:44.942818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:44.943113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:44.970551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] MFA Rules not processed for user `c8b735eb05fa417880f159c384b0b4d7`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:44.972009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:44.988270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:DELETE=1,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:45.035997 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1252/2484] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:44 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:45.041107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.052417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.054585 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:45.062529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:45.114614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f06f88ad-e57d-47ba-bcc3-16e87fa7f92c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.120191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.120461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.120738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.121006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.121656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.123593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.124488 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf12218-684d-4885-8d1a-dce0401994db None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.129522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.130142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.130416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.130678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] PATH_INFO: `/v3/projects/6e6a916acbec45469085afadc02a8647` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.131930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e6a916acbec45469085afadc02a8647)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.133737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.220456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1233/2485] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/projects => generated 373 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:45.226311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.260559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e6a916acbec45469085afadc02a8647', 'request_id': 'req-5d753fa6-1452-4497-8990-a937f99411b9'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:45.290184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5092d14c-5155-4718-a9ec-6361a4abe3d5 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.344870 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.345671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.345768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.345994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.346619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.348601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.350348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e6a916acbec45469085afadc02a8647', 'request_id': 'req-5d753fa6-1452-4497-8990-a937f99411b9'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:45.358018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e6a916acbec45469085afadc02a8647', 'request_id': 'req-5d753fa6-1452-4497-8990-a937f99411b9'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:45.368912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] The token cache is being invalidate because project 6e6a916acbec45469085afadc02a8647 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:45.369203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:45.369866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d753fa6-1452-4497-8990-a937f99411b9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:45.371402 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1253/2486] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:45 2022] DELETE /identity/v3/projects/6e6a916acbec45469085afadc02a8647 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:45.376146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a790d638-c0d5-4fab-9633-82c0d19beb35 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.376424 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a790d638-c0d5-4fab-9633-82c0d19beb35 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.376693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a790d638-c0d5-4fab-9633-82c0d19beb35 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.395920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a790d638-c0d5-4fab-9633-82c0d19beb35 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:45.397135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a790d638-c0d5-4fab-9633-82c0d19beb35 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:45.406427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1254/2487] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:45.415392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.415669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.415940 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.435093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:45.436281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:45.444227 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1255/2488] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:45.448388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.454102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1234/2489] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/users => generated 293 bytes in 226 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:45.455219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.457820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0fa1821-e5f5-44b4-91e5-723191f15b61 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.458053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:45.463371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.463962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.464234 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.464497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.465165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:43:45.465461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.467157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.470961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95a03cc4-3a28-4ec4-9c06-585dcb545c9f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.477059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.477844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.478124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.478396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.479131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.479907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1256/2490] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:43:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:43:45.484071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.487740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.492776 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1235/2491] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:45 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:45.496929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.502134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d764a8b-3eef-429b-b6fa-5e7634e26953 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.507429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.507734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-073d1e32-8de6-4ef8-875a-bcb8ad3c4253 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.508020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.508297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.509194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.509833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.511887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.512549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.513260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.513605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.513957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] PATH_INFO: `/v3/projects/c30761d86cb748b9a1b02e3cad81b8bc/users/b008fb898ec54c0d99b7f078a8b38d6d/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.531512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c30761d86cb748b9a1b02e3cad81b8bc, user_id=b008fb898ec54c0d99b7f078a8b38d6d, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.534167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.634330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:45.640798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1257/2492] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/projects => generated 391 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:45.644770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.650662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1236/2493] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:45 2022] PUT /identity/v3/projects/c30761d86cb748b9a1b02e3cad81b8bc/users/b008fb898ec54c0d99b7f078a8b38d6d/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:45.654885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.660410 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:45.709907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cafd5415-5bd0-4144-a91c-ae114ffdb799 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.714986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.715587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.715867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.716137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.716791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.716887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b00247d-47ad-4c51-930c-c90abc6926a7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.718767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.722499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.723138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.723452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.723745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.724380 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.726477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.736708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1237/2494] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:45 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:43:45.741036 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.751803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99bbcbdd-dc91-4f92-b506-f6cc1fc0e655 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.756474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-746ed365-5da4-4658-b29a-d7ff3d864d0f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.757466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746ed365-5da4-4658-b29a-d7ff3d864d0f admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.757742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746ed365-5da4-4658-b29a-d7ff3d864d0f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.758017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-746ed365-5da4-4658-b29a-d7ff3d864d0f admin admin] PATH_INFO: `/v3/projects/c30761d86cb748b9a1b02e3cad81b8bc/users/b008fb898ec54c0d99b7f078a8b38d6d/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.759984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746ed365-5da4-4658-b29a-d7ff3d864d0f admin admin] RBAC: Authorizing `identity:create_grant(project_id=c30761d86cb748b9a1b02e3cad81b8bc, user_id=b008fb898ec54c0d99b7f078a8b38d6d, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.762730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-746ed365-5da4-4658-b29a-d7ff3d864d0f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.774135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1238/2495] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:45 2022] PUT /identity/v3/projects/c30761d86cb748b9a1b02e3cad81b8bc/users/b008fb898ec54c0d99b7f078a8b38d6d/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:45.784513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.784798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.785075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.805894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1258/2496] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/users => generated 296 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:45.807269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.814501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] MFA Rules not processed for user `b008fb898ec54c0d99b7f078a8b38d6d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:45.824722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:45.824862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce82abaf-e9ef-4c66-9637-1bc97d04272d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.876083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.876706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.876957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.877228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.877894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.879525 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.891196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1239/2497] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:45 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:45.891396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1259/2498] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:45 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:45.893924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.900409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.902090 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:45.904614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f283f28-62ef-496d-b5a7-e3c4ef576fb0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.909124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.909716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.910597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.910913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.911551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.913635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.923811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1240/2499] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:45 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:45.927912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:45.938571 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48110804-a66b-4034-8b8c-20ad01f4d8c7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.943240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.943872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.944175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.944821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] PATH_INFO: `/v3/projects/f9c5c3a6ec894c5dbe50777134d0ad3f/users/6b824d30b0ba4bf0a7282cd980e545dc/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.959427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f9c5c3a6ec894c5dbe50777134d0ad3f, user_id=6b824d30b0ba4bf0a7282cd980e545dc, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.962372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84575fae-52f5-424c-9152-7f0bd5dfa22d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:45.963429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.965987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34a67be8-9ebe-4428-ae24-a483cc93b532 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:45.966581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a67be8-9ebe-4428-ae24-a483cc93b532 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:45.966888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a67be8-9ebe-4428-ae24-a483cc93b532 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:45.967155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34a67be8-9ebe-4428-ae24-a483cc93b532 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:45.977361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a67be8-9ebe-4428-ae24-a483cc93b532 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:45.978911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34a67be8-9ebe-4428-ae24-a483cc93b532 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:45.988358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1260/2500] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:45 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:43:46.039117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1241/2501] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:45 2022] PUT /identity/v3/projects/f9c5c3a6ec894c5dbe50777134d0ad3f/users/6b824d30b0ba4bf0a7282cd980e545dc/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:46.051653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.051933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.052241 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.079352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] MFA Rules not processed for user `6b824d30b0ba4bf0a7282cd980e545dc`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:46.080550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:46.097013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:46.143924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1261/2502] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:46 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:46.149216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.190694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:46.212288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74b51eec-cd81-4ffd-980a-4b5926058080 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.216957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.217590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.217865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.218134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.218750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.220676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.246701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.301609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a570760a-c3f2-4f2c-817f-11036026a835 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.305319 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.305892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.311202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.311202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.311202 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:46.338719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1242/2503] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:46 2022] POST /identity/v3/projects => generated 391 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:46.347140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.368133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.369806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.379396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1262/2504] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:46 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:43:46.412076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffd19fab-c6eb-44b9-aaec-d6e924dbf16d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.417095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.417774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.418129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.418404 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.419061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.421080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.508146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1243/2505] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:46 2022] POST /identity/v3/users => generated 302 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:43:46.512211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.523885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e50fa56-cc11-4a67-93c1-039275639a10 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.528828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.529450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.529726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.529996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.530632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.532239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.542540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1263/2506] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:46 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:46.546847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.556893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3484f711-a87b-479c-aa8e-e2faf632f135 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.561766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.562418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.562695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.563008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] PATH_INFO: `/v3/projects/255df5736f914cd1b1fadc35061b58f4/users/9f3c5837c8fb43a482379a57112b24aa/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.578684 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=255df5736f914cd1b1fadc35061b58f4, user_id=9f3c5837c8fb43a482379a57112b24aa, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.581415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.714257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1244/2507] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:46 2022] PUT /identity/v3/projects/255df5736f914cd1b1fadc35061b58f4/users/9f3c5837c8fb43a482379a57112b24aa/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:46.718155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.728267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c5231b-72ee-47e1-8f54-72b9fee2e1c8 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.791071 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a631298-8848-434e-8ab2-bd5298fc18f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.791667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a631298-8848-434e-8ab2-bd5298fc18f7 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.791946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a631298-8848-434e-8ab2-bd5298fc18f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.792212 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a631298-8848-434e-8ab2-bd5298fc18f7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.792839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a631298-8848-434e-8ab2-bd5298fc18f7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.794459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a631298-8848-434e-8ab2-bd5298fc18f7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.796461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:46.805034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1264/2508] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:46 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:46.809751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:46.821090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c73afcc-383d-408e-8af8-f9c4cf384cf1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:46.825712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:46.826356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.826633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.826944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] PATH_INFO: `/v3/projects/255df5736f914cd1b1fadc35061b58f4/users/9f3c5837c8fb43a482379a57112b24aa/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.828839 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] RBAC: Authorizing `identity:create_grant(project_id=255df5736f914cd1b1fadc35061b58f4, user_id=9f3c5837c8fb43a482379a57112b24aa, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:46.831558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:46.841928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1245/2509] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:46 2022] PUT /identity/v3/projects/255df5736f914cd1b1fadc35061b58f4/users/9f3c5837c8fb43a482379a57112b24aa/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:46.852792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:46.857186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:46.857186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:46.886554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] MFA Rules not processed for user `9f3c5837c8fb43a482379a57112b24aa`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:46.887891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:46.958073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1265/2510] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:46 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:46.969192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:47.028939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96079cc0-c438-475d-b8a2-9d2e254c09ef admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:47.032686 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:47.033267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:47.033544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:47.033813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:47.044177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:47.046238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:47.058429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1246/2511] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:46 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:47.109269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:47.110735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:47.123668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f11b114-58a5-4819-9a18-6e17fb14923b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:47.127452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50d123f-5383-462c-ac60-ca97dbdca789 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:47.138283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50d123f-5383-462c-ac60-ca97dbdca789 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:47.138283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50d123f-5383-462c-ac60-ca97dbdca789 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:47.138283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50d123f-5383-462c-ac60-ca97dbdca789 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:47.185471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50d123f-5383-462c-ac60-ca97dbdca789 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:47.186921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50d123f-5383-462c-ac60-ca97dbdca789 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:47.195774 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1266/2512] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:47 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:43:55.308881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:55.327726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:55.373664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcb6f980-f482-48b7-ae68-0d07bf2ea9be service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:55.378213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:55.378875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:55.379158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:55.379436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] PATH_INFO: `/v3/users/6b824d30b0ba4bf0a7282cd980e545dc` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:55.380554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=6b824d30b0ba4bf0a7282cd980e545dc)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:55.382491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:55.581439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6b824d30b0ba4bf0a7282cd980e545dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:55.589170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6b824d30b0ba4bf0a7282cd980e545dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:55.589852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6b824d30b0ba4bf0a7282cd980e545dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:55.590448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6b824d30b0ba4bf0a7282cd980e545dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:55.661893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6b824d30b0ba4bf0a7282cd980e545dc'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:55.685513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1247/2513] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:55 2022] DELETE /identity/v3/users/6b824d30b0ba4bf0a7282cd980e545dc => generated 0 bytes in 377 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:55.690143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:55.690645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:55.690871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:55.716830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:55.717743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:55.795563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1267/2514] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:43:55.805869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:55.807487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:55.868481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0828df35-5724-4845-999c-cb64c5ce14bb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:55.872197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:55.872801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:55.873079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:55.873351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:55.882467 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:55.884174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:55.893757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1248/2515] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:55 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:43:56.315185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:56.317052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:56.353611 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7bb1ae-15b6-42ac-889b-4eb61d942dde None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:56.359166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:56.359811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:56.361442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:56.361442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] PATH_INFO: `/v3/projects/f9c5c3a6ec894c5dbe50777134d0ad3f` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:56.361639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f9c5c3a6ec894c5dbe50777134d0ad3f)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:56.363539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:56.383113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:56.478308 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f9c5c3a6ec894c5dbe50777134d0ad3f', 'request_id': 'req-21c17155-2a2d-4f17-9331-543f32f844d1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.556490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f9c5c3a6ec894c5dbe50777134d0ad3f', 'request_id': 'req-21c17155-2a2d-4f17-9331-543f32f844d1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.563669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f9c5c3a6ec894c5dbe50777134d0ad3f', 'request_id': 'req-21c17155-2a2d-4f17-9331-543f32f844d1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.574935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] The token cache is being invalidate because project f9c5c3a6ec894c5dbe50777134d0ad3f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:56.575161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.575806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.577290 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1268/2516] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:56 2022] DELETE /identity/v3/projects/f9c5c3a6ec894c5dbe50777134d0ad3f => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:56.581244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:56.591164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaf94e5-1dc5-4618-b7db-b4ecc509c6a1 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:56.630573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:56.649318 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:56.650533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:56.650845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:56.651124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] PATH_INFO: `/v3/users/9f3c5837c8fb43a482379a57112b24aa` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:56.652274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9f3c5837c8fb43a482379a57112b24aa)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:56.654573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:56.875831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9f3c5837c8fb43a482379a57112b24aa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.883829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9f3c5837c8fb43a482379a57112b24aa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.885244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9f3c5837c8fb43a482379a57112b24aa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.885963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9f3c5837c8fb43a482379a57112b24aa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:56.990067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8b071b5d-47a3-4af7-9d40-ac8b19108095 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9f3c5837c8fb43a482379a57112b24aa'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:57.014568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1249/2517] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:43:56 2022] DELETE /identity/v3/users/9f3c5837c8fb43a482379a57112b24aa => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:43:57.666278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:57.667768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:57.688292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21c17155-2a2d-4f17-9331-543f32f844d1 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:57.755322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:57.755322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:57.755322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:57.755322 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] PATH_INFO: `/v3/projects/255df5736f914cd1b1fadc35061b58f4` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:57.755712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] RBAC: Authorizing `identity:delete_project(project_id=255df5736f914cd1b1fadc35061b58f4)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:57.757273 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:57.952148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '255df5736f914cd1b1fadc35061b58f4', 'request_id': 'req-57f8f40a-a211-4d2b-b8a0-b6307428cc65'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:58.088084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '255df5736f914cd1b1fadc35061b58f4', 'request_id': 'req-57f8f40a-a211-4d2b-b8a0-b6307428cc65'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:58.095990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '255df5736f914cd1b1fadc35061b58f4', 'request_id': 'req-57f8f40a-a211-4d2b-b8a0-b6307428cc65'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:58.107124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] The token cache is being invalidate because project 255df5736f914cd1b1fadc35061b58f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:43:58.107410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:58.108054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-57f8f40a-a211-4d2b-b8a0-b6307428cc65 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:43:58.109539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1269/2518] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:43:57 2022] DELETE /identity/v3/projects/255df5736f914cd1b1fadc35061b58f4 => generated 0 bytes in 444 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:58.114434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.114964 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.115161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.136224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:58.137476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:58.207461 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1250/2519] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:58.215629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.215978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.216233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.239186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:58.240883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:58.249947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1270/2520] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:58.254305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:58.268527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d17b7f80-6581-442e-97f4-4650fb1cdeb7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:58.274838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:58.276093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.276377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.276654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.277336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:43:58.277643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:58.279576 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:58.292944 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1251/2521] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:43:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:43:58.297442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:58.307089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc6819f2-d06d-4757-8134-cd71c0560cdc None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:58.311826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:58.312417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.312695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.312966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.313593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:58.315579 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:58.429262 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1271/2522] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:58 2022] POST /identity/v3/projects => generated 389 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:58.433683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:58.443973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b042505-a54a-4117-ac26-7249569e5fb7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:58.492486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:58.505362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:58.505970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.506252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.506526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.507186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:58.509617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:58.616093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1252/2523] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:58 2022] POST /identity/v3/users => generated 295 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:58.620449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:58.630757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-065fc70f-efd3-4359-94f0-a9a7bdd1be99 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:58.636258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:58.636848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.637126 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.641377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.641377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:58.641377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:58.641377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:58.650678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1272/2524] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:58.654892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:58.664710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d7a8281-3f8e-4b22-aaae-abcea855a779 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:58.669391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d92ad7f6-7ba1-4c21-acbe-abe8453bedc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:58.669984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ad7f6-7ba1-4c21-acbe-abe8453bedc1 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.670259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ad7f6-7ba1-4c21-acbe-abe8453bedc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.670527 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d92ad7f6-7ba1-4c21-acbe-abe8453bedc1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.671188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92ad7f6-7ba1-4c21-acbe-abe8453bedc1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:58.672852 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d92ad7f6-7ba1-4c21-acbe-abe8453bedc1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:58.684667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1253/2525] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:58.688933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:58.699476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c835218-1f47-4fa6-96b5-0b3857d1199d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:58.704252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:58.704922 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.705199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.705492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] PATH_INFO: `/v3/projects/fff9c8b2b5a84936b702f22dbdac54a1/users/49c72698cbed44bab0c018da5d615d74/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.719848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fff9c8b2b5a84936b702f22dbdac54a1, user_id=49c72698cbed44bab0c018da5d615d74, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:58.722581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:58.839785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1273/2526] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:58 2022] PUT /identity/v3/projects/fff9c8b2b5a84936b702f22dbdac54a1/users/49c72698cbed44bab0c018da5d615d74/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:58.890938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:58.891226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:58.891736 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:58.923286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] MFA Rules not processed for user `49c72698cbed44bab0c018da5d615d74`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:58.924554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:58.992790 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1254/2527] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:58 2022] POST /identity/v3/auth/tokens => generated 3266 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:58.997980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.063192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eacb1064-1a91-472b-b8fe-48911db59ae4 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.067885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.068487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.068887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.069161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.069792 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.072078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.189052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1274/2528] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:43:58 2022] POST /identity/v3/projects => generated 387 bytes in 188 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:59.189993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.201053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-935505f5-cd90-4a25-bb37-40edafbbde3d None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.261600 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.262167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.262449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.263361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.264032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.265956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.411505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1255/2529] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:43:59 2022] POST /identity/v3/users => generated 300 bytes in 222 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:43:59.418707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.429602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e2906ea-2e89-4e7c-b5ed-0d03ff3c5a0b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.435039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.435659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.437276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.437584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.438224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.449787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.463609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1275/2530] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:59 2022] GET /identity/v3/roles => generated 1420 bytes in 45 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:59.467770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.478455 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc23fd5f-72d4-454f-a3f3-6c3b97a5f272 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.483357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.484019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.484296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.484568 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] PATH_INFO: `/v3/projects/9e10740262994999b73b23e7a39a8f30/users/b2207ae43c4548b3a914b571547c9dc1/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.500175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e10740262994999b73b23e7a39a8f30, user_id=b2207ae43c4548b3a914b571547c9dc1, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.502984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.614726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1256/2531] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:59 2022] PUT /identity/v3/projects/9e10740262994999b73b23e7a39a8f30/users/b2207ae43c4548b3a914b571547c9dc1/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:59.618463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.633617 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99df457d-f544-46e8-beba-4f9059868d0e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.689219 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc20b6b9-8493-46fc-84d5-accd135f2bea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.689927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc20b6b9-8493-46fc-84d5-accd135f2bea admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.689927 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc20b6b9-8493-46fc-84d5-accd135f2bea admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.690194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc20b6b9-8493-46fc-84d5-accd135f2bea admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.690865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc20b6b9-8493-46fc-84d5-accd135f2bea admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.692468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc20b6b9-8493-46fc-84d5-accd135f2bea admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.702955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1276/2532] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:43:59 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:43:59.707133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.718464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1ca485-96b8-4197-8dc8-fee1ef5e8766 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.723210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.723861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.724136 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.724415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] PATH_INFO: `/v3/projects/9e10740262994999b73b23e7a39a8f30/users/b2207ae43c4548b3a914b571547c9dc1/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.726377 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9e10740262994999b73b23e7a39a8f30, user_id=b2207ae43c4548b3a914b571547c9dc1, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.729725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.740822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1257/2533] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:43:59 2022] PUT /identity/v3/projects/9e10740262994999b73b23e7a39a8f30/users/b2207ae43c4548b3a914b571547c9dc1/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:43:59.751895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.752190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.752472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.781946 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] MFA Rules not processed for user `b2207ae43c4548b3a914b571547c9dc1`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:43:59.783233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:59.813573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:59.850506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1277/2534] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:43:59 2022] POST /identity/v3/auth/tokens => generated 3331 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:43:59.862163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:43:59.866974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:43:59.877630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:43:59.930441 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f691c6fa-f85c-49e1-94f9-2f619d1a1db7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:43:59.940721 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fdbf386-1d5c-4c8f-9b8b-cbe61c2a4b63 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:43:59.941219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbf386-1d5c-4c8f-9b8b-cbe61c2a4b63 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:43:59.941495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbf386-1d5c-4c8f-9b8b-cbe61c2a4b63 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:43:59.941763 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fdbf386-1d5c-4c8f-9b8b-cbe61c2a4b63 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:43:59.952711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbf386-1d5c-4c8f-9b8b-cbe61c2a4b63 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:43:59.954277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fdbf386-1d5c-4c8f-9b8b-cbe61c2a4b63 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:43:59.967336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1258/2535] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:43:59 2022] GET /identity/v3/auth/tokens => generated 3331 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:00.345249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:00.415544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15c629ca-b975-4c8f-a774-b3d39cdfdab8 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:00.420421 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:00.421058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:00.421335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:00.421610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] PATH_INFO: `/v3/users/49c72698cbed44bab0c018da5d615d74` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:00.422732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=49c72698cbed44bab0c018da5d615d74)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:00.424690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:00.678664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '49c72698cbed44bab0c018da5d615d74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:00.679392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '49c72698cbed44bab0c018da5d615d74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:00.687156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '49c72698cbed44bab0c018da5d615d74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:00.687795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '49c72698cbed44bab0c018da5d615d74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:00.801647 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '49c72698cbed44bab0c018da5d615d74'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:00.820639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1278/2536] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:00 2022] DELETE /identity/v3/users/49c72698cbed44bab0c018da5d615d74 => generated 0 bytes in 476 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:00.825431 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:00.825923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:00.826038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:00.844807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:00.846040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:00.905644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1259/2537] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:00.915954 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:00.922094 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:00.970878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:00.975975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90c6d859-4072-4411-88b6-1ebd68025ee0 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:00.980303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:00.980907 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:00.981182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:00.983330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:00.990245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:00.991754 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:01.001173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1279/2538] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:01.442758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:01.444211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:01.457939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938e72b2-fafc-40f3-b266-d453c9da958b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:01.470909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:01.470909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:01.470909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:01.470909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] PATH_INFO: `/v3/projects/fff9c8b2b5a84936b702f22dbdac54a1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:01.471445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fff9c8b2b5a84936b702f22dbdac54a1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:01.472590 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:01.582409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fff9c8b2b5a84936b702f22dbdac54a1', 'request_id': 'req-bec672c7-a499-4b60-a435-ecaf35107b75'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.588762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fff9c8b2b5a84936b702f22dbdac54a1', 'request_id': 'req-bec672c7-a499-4b60-a435-ecaf35107b75'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.663978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fff9c8b2b5a84936b702f22dbdac54a1', 'request_id': 'req-bec672c7-a499-4b60-a435-ecaf35107b75'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.675948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] The token cache is being invalidate because project fff9c8b2b5a84936b702f22dbdac54a1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:01.676234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.676861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.678397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1260/2539] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:01 2022] DELETE /identity/v3/projects/fff9c8b2b5a84936b702f22dbdac54a1 => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:01.683060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:01.692546 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66450ef0-017b-4229-b7bc-c9b3a6a48bce service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:01.749757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:01.750395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:01.750673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:01.750968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] PATH_INFO: `/v3/users/b2207ae43c4548b3a914b571547c9dc1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:01.752109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b2207ae43c4548b3a914b571547c9dc1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:01.754156 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:01.963653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b2207ae43c4548b3a914b571547c9dc1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.964353 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b2207ae43c4548b3a914b571547c9dc1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.971108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b2207ae43c4548b3a914b571547c9dc1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:01.971723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b2207ae43c4548b3a914b571547c9dc1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.044673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b2207ae43c4548b3a914b571547c9dc1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.063565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1280/2540] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:01 2022] DELETE /identity/v3/users/b2207ae43c4548b3a914b571547c9dc1 => generated 0 bytes in 381 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:02.594620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:02.596109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:02.611003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec672c7-a499-4b60-a435-ecaf35107b75 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:02.645078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:02.671675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:02.672299 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:02.672573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:02.672840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] PATH_INFO: `/v3/projects/9e10740262994999b73b23e7a39a8f30` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:02.674004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] RBAC: Authorizing `identity:delete_project(project_id=9e10740262994999b73b23e7a39a8f30)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:02.675954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:02.778720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:02.779975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:02.836943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dfcdb999-4da5-41e5-8dbb-05755c39e3c5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:02.841223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:02.841826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:02.842101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:02.842371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:02.844101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:02.856548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:02.858064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-329d27f8-fab5-4e1f-af53-77d212141601 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:02.868430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1281/2541] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:02 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:44:02.873949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9e10740262994999b73b23e7a39a8f30', 'request_id': 'req-4efffebf-97d8-4533-9e9f-48de6575b53a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.874548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9e10740262994999b73b23e7a39a8f30', 'request_id': 'req-4efffebf-97d8-4533-9e9f-48de6575b53a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.943931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9e10740262994999b73b23e7a39a8f30', 'request_id': 'req-4efffebf-97d8-4533-9e9f-48de6575b53a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.957047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] The token cache is being invalidate because project 9e10740262994999b73b23e7a39a8f30 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:02.957446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.958088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4efffebf-97d8-4533-9e9f-48de6575b53a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:02.959797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1261/2542] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:02 2022] DELETE /identity/v3/projects/9e10740262994999b73b23e7a39a8f30 => generated 0 bytes in 366 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:02.965345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:02.965874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:02.965974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:02.985433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:02.986632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:02.991923 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:03.049420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1282/2543] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:03.057699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.058363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.058636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.081173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:03.082391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:03.091185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1262/2544] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:03 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:03.095654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.105812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ed51057-0063-46e3-a006-e769a19cfa13 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.110433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.111057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.111335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.111607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.112251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:44:03.112555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.114145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.126668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1283/2545] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:44:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:44:03.131086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.141288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bedff0b6-510e-4e37-aa61-032f3992a808 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.146025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.146631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.146970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.147261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.147903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.149851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.239632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1263/2546] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:03 2022] POST /identity/v3/projects => generated 409 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:44:03.240866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.251654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff39a9b-9dfd-4837-870b-c20b7f0dc77b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.308965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.309577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.309854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.310127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.310793 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.312853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.396124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1284/2547] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:03 2022] POST /identity/v3/users => generated 305 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:03.400189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.419090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c802526-4c17-45e5-b913-eb5ab9700f71 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.425516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.426128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.426409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.426737 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.427477 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.429075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.444370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1264/2548] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:03 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:03.445456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.456456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd871f66-86b7-4210-a65e-2791335afbba admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.462177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ed31e78-786d-4cc6-8c17-8d7f837b98ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.462917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed31e78-786d-4cc6-8c17-8d7f837b98ca admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.463374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed31e78-786d-4cc6-8c17-8d7f837b98ca admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.464166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ed31e78-786d-4cc6-8c17-8d7f837b98ca admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.464498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed31e78-786d-4cc6-8c17-8d7f837b98ca admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.466302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ed31e78-786d-4cc6-8c17-8d7f837b98ca admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.478808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1285/2549] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:03 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:03.482850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.493216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c8b599b-61bc-4161-ad5e-4b1f6fd8727d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.499547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.500211 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.500490 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.500775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] PATH_INFO: `/v3/projects/f23f275251304aff8e92cb0c1d112719/users/e24a3c489e154a67a96fb47e2e5bfce2/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.516788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] RBAC: Authorizing `identity:create_grant(project_id=f23f275251304aff8e92cb0c1d112719, user_id=e24a3c489e154a67a96fb47e2e5bfce2, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.519679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.611158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1265/2550] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:03 2022] PUT /identity/v3/projects/f23f275251304aff8e92cb0c1d112719/users/e24a3c489e154a67a96fb47e2e5bfce2/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:44:03.623802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.624068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.624335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.657173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] MFA Rules not processed for user `e24a3c489e154a67a96fb47e2e5bfce2`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:03.658537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:03.725838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1286/2551] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:03 2022] POST /identity/v3/auth/tokens => generated 3286 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:03.731698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.797720 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e30a5a-09fe-446f-bc9c-97974042b08e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.802548 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:03.803184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:03.803460 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:03.803731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:03.804403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:03.807084 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:03.822602 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:03.929592 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1266/2552] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:03 2022] POST /identity/v3/projects => generated 409 bytes in 199 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:03.934041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:03.945063 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce9aba8-1682-4c75-9245-bbe7279e6ebd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:03.949572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:04.011955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:04.014990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.014990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.014990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.014990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:04.016004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:04.135663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1287/2553] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:03 2022] POST /identity/v3/users => generated 311 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:44:04.138901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:04.154532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2026ab-8abf-4087-a5aa-e8186c21265e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:04.165623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:04.166903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.167129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.167365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.168016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:04.171881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:04.195300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1267/2554] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:04 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:04.205425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:04.217432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c410c2db-09e6-4116-acd6-d07ad31f35f5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:04.222150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:04.222972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.223255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.223527 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] PATH_INFO: `/v3/projects/9adda77e3aad43a78dbeb5c8d9c59178/users/e1e4d248d53b42158e554d01ba586d9c/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.243048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9adda77e3aad43a78dbeb5c8d9c59178, user_id=e1e4d248d53b42158e554d01ba586d9c, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:04.246067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:04.343014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1288/2555] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:04 2022] PUT /identity/v3/projects/9adda77e3aad43a78dbeb5c8d9c59178/users/e1e4d248d53b42158e554d01ba586d9c/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:04.344163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:04.355835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99f91a3a-554b-43eb-a714-e4d527dc47c7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:04.415295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a6b9031-4554-4854-9452-5b78ebe76e55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:04.415872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6b9031-4554-4854-9452-5b78ebe76e55 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.416147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6b9031-4554-4854-9452-5b78ebe76e55 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.416478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a6b9031-4554-4854-9452-5b78ebe76e55 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.417106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6b9031-4554-4854-9452-5b78ebe76e55 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:04.418709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a6b9031-4554-4854-9452-5b78ebe76e55 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:04.428633 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1268/2556] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:04 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:44:04.433803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:04.445881 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6216533-b8e9-4e6d-bf0e-d89df797c4bc admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:04.453120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:04.453853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.454130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.454817 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] PATH_INFO: `/v3/projects/9adda77e3aad43a78dbeb5c8d9c59178/users/e1e4d248d53b42158e554d01ba586d9c/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.456388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] RBAC: Authorizing `identity:create_grant(project_id=9adda77e3aad43a78dbeb5c8d9c59178, user_id=e1e4d248d53b42158e554d01ba586d9c, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:04.459618 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:04.471601 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1289/2557] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:04 2022] PUT /identity/v3/projects/9adda77e3aad43a78dbeb5c8d9c59178/users/e1e4d248d53b42158e554d01ba586d9c/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:04.482524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.482848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.483135 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.513887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] MFA Rules not processed for user `e1e4d248d53b42158e554d01ba586d9c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:04.515195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:04.590803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1269/2558] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:04 2022] POST /identity/v3/auth/tokens => generated 3353 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:04.608746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:04.610694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:04.686885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c3aad18-176c-4e31-9d00-8d5c2474e6cb admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:04.690996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:04.691739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:04.692135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:04.692496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:04.702680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:04.704324 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:04.715675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1290/2559] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:04 2022] GET /identity/v3/auth/tokens => generated 3353 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:05.089409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:05.099824 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7917e7ed-89fe-4a2b-9d5a-0fd5cf7ce551 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:05.106190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:05.106927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:05.107281 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:05.107726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:05.170389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:05.174596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:05.176428 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:05.195509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1270/2560] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:05 2022] GET /identity/v3/auth/tokens => generated 3286 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:06.936355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:06.937906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:07.004112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60f3a482-74a1-4511-a28f-19ab46fa61b4 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:07.008775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:07.010188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:07.010307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:07.010554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] PATH_INFO: `/v3/users/1dbfb1a3e99c418098c2b02aea93b923` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:07.011761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=1dbfb1a3e99c418098c2b02aea93b923)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:07.013685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:07.016573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:07.314150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1dbfb1a3e99c418098c2b02aea93b923'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.315004 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1dbfb1a3e99c418098c2b02aea93b923'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.322012 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1dbfb1a3e99c418098c2b02aea93b923'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.322653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1dbfb1a3e99c418098c2b02aea93b923'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.416658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c742cd92-5bc8-4c1c-81bb-87e66d4cf2ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1dbfb1a3e99c418098c2b02aea93b923'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.436728 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1291/2561] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:06 2022] DELETE /identity/v3/users/1dbfb1a3e99c418098c2b02aea93b923 => generated 0 bytes in 501 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:07.675338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:07.687646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5384a254-4ded-415f-90ef-67063c1db0d0 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:07.746149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:07.746808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:07.747131 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:07.747402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] PATH_INFO: `/v3/projects/41cca09950c34cc7baded0eb0a42bb07` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:07.748652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] RBAC: Authorizing `identity:delete_project(project_id=41cca09950c34cc7baded0eb0a42bb07)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:07.750611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:07.881100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '41cca09950c34cc7baded0eb0a42bb07', 'request_id': 'req-b037de0e-6ff8-4f09-aad1-7c8381830186'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.886970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '41cca09950c34cc7baded0eb0a42bb07', 'request_id': 'req-b037de0e-6ff8-4f09-aad1-7c8381830186'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.960936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '41cca09950c34cc7baded0eb0a42bb07', 'request_id': 'req-b037de0e-6ff8-4f09-aad1-7c8381830186'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.973260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] The token cache is being invalidate because project 41cca09950c34cc7baded0eb0a42bb07 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:07.973545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.974235 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b037de0e-6ff8-4f09-aad1-7c8381830186 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:07.975731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1271/2562] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:07 2022] DELETE /identity/v3/projects/41cca09950c34cc7baded0eb0a42bb07 => generated 0 bytes in 301 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:07.981368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:07.981876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:07.981986 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.001433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:08.002620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:08.062720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1292/2563] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:08.070552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.070862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.071177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.092176 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:08.093379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:08.101381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1272/2564] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:08.105540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.115125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b11f7f9b-9398-4635-8c56-ab45e06debf9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.119956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.120552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.122870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.122870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.122870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:44:08.122870 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.124259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.136770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1293/2565] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:44:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:44:08.141155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.151026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0cb098e-b5f7-476c-af0c-f359a90a2b3c None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.155965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.156554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.156833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.157102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.157732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.159677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.241868 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1273/2566] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:08 2022] POST /identity/v3/projects => generated 377 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:08.246091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.256167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1611df-b39c-4b23-8686-503e37d1bc21 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.311956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.312602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.312880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.313150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.313781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.315884 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.430050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:08.436685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1294/2567] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:08 2022] POST /identity/v3/users => generated 289 bytes in 191 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:08.440953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.451864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b3e6c23-5d7e-4d75-bee2-fce3e906affe admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.456514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.457112 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.457391 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.457664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.458301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.459955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.469906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1274/2568] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:08 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:08.473910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.484173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-353b74dc-88fd-4c1f-8a1c-c34f435b8318 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.489634 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd242756-c411-4a3b-a8b1-d98b14ae0c15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.490246 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd242756-c411-4a3b-a8b1-d98b14ae0c15 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.490550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd242756-c411-4a3b-a8b1-d98b14ae0c15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.490855 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd242756-c411-4a3b-a8b1-d98b14ae0c15 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.491536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd242756-c411-4a3b-a8b1-d98b14ae0c15 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.493209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd242756-c411-4a3b-a8b1-d98b14ae0c15 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.505411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1295/2569] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:08.510886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.520887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b2defa-2bfe-404e-a0f5-4eb1d3112d48 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.526030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.526755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.527074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.534951 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] PATH_INFO: `/v3/projects/0a89110c26344f608dae59f5616575f3/users/09cb0d1d7888436dae18a45598a8c82a/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.542688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] RBAC: Authorizing `identity:create_grant(project_id=0a89110c26344f608dae59f5616575f3, user_id=09cb0d1d7888436dae18a45598a8c82a, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.545393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.658470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1275/2570] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:08 2022] PUT /identity/v3/projects/0a89110c26344f608dae59f5616575f3/users/09cb0d1d7888436dae18a45598a8c82a/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:44:08.663776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbe0c8-1cc8-46da-91bb-b0f7378ae5b3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.664066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbe0c8-1cc8-46da-91bb-b0f7378ae5b3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.664370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddfbe0c8-1cc8-46da-91bb-b0f7378ae5b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.693730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ddfbe0c8-1cc8-46da-91bb-b0f7378ae5b3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:08.693730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ddfbe0c8-1cc8-46da-91bb-b0f7378ae5b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:08.748067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1296/2571] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:08 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:08.757632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:08.762359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:08.783081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:08.819368 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a724612a-c8c3-4459-bb39-47a6b9548010 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:08.823657 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:08.824263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:08.824545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:08.824820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:08.833892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:08.835369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:08.845312 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1276/2572] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:14.416851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cc6da6-cb2a-4af1-9f8f-cbeac03a82d8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:14.417894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cc6da6-cb2a-4af1-9f8f-cbeac03a82d8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:14.418418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34cc6da6-cb2a-4af1-9f8f-cbeac03a82d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:14.449396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-34cc6da6-cb2a-4af1-9f8f-cbeac03a82d8 None None] MFA Rules not processed for user `09cb0d1d7888436dae18a45598a8c82a`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:14.450769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-34cc6da6-cb2a-4af1-9f8f-cbeac03a82d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:14.524381 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1297/2573] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:14 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:14.540586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:14.542038 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:14.623546 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4314c6b-cd0a-43a3-9951-eb8009f45f6d service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:14.629068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:14.629683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:14.630020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:14.630293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:14.639423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:14.641014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:14.650762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1277/2574] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:14 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:17.076950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:17.077636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:17.077636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:17.100223 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] MFA Rules not processed for user `09cb0d1d7888436dae18a45598a8c82a`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:17.101480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:17.110023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1298/2575] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:17 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:17.496741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:17.498242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:17.515004 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb53546a-ed9e-4543-a273-65f5aebf7b36 service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:17.519336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:17.519981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:17.520481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:17.520922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:17.523223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:17.535263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:17.537366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d76fb89-5734-4e36-a15f-0cd78b4e91a9 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:17.548083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1278/2576] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:17 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:20.030019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:20.031556 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:20.045549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cae745e2-6445-4d7b-9e8c-df443f4cd55e None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:20.050081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:20.054089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:20.054089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:20.054089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] PATH_INFO: `/v3/users/e24a3c489e154a67a96fb47e2e5bfce2` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:20.054089 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] RBAC: Authorizing `identity:delete_user(user_id=e24a3c489e154a67a96fb47e2e5bfce2)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:20.054470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:20.239238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e24a3c489e154a67a96fb47e2e5bfce2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:20.240016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e24a3c489e154a67a96fb47e2e5bfce2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:20.249347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e24a3c489e154a67a96fb47e2e5bfce2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:20.249943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e24a3c489e154a67a96fb47e2e5bfce2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:20.354713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e24a3c489e154a67a96fb47e2e5bfce2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:20.357726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:20.374128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1299/2577] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:20 2022] DELETE /identity/v3/users/e24a3c489e154a67a96fb47e2e5bfce2 => generated 0 bytes in 345 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:20.379219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:20.380053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:20.380429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:20.399981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:20.401180 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:20.467917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1279/2578] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:20 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:20.479795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:20.481963 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:20.542720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67e5df71-4143-463a-a187-7f139e1cf6ed admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:20.547318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:20.548153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:20.548516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:20.548835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:20.558598 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:20.560411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:20.570271 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1300/2579] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:20 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:21.131893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:21.133222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:21.146196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c186dd10-5687-439b-91d2-40ad2ec579a9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:21.151781 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:21.154223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:21.154545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:21.154962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] PATH_INFO: `/v3/projects/f23f275251304aff8e92cb0c1d112719` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:21.156203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f23f275251304aff8e92cb0c1d112719)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:21.158272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:21.263141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:21.276417 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f23f275251304aff8e92cb0c1d112719', 'request_id': 'req-8a7f21c1-f802-456d-979f-2be7c35612a6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.283184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f23f275251304aff8e92cb0c1d112719', 'request_id': 'req-8a7f21c1-f802-456d-979f-2be7c35612a6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.392022 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f23f275251304aff8e92cb0c1d112719', 'request_id': 'req-8a7f21c1-f802-456d-979f-2be7c35612a6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.406057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] The token cache is being invalidate because project f23f275251304aff8e92cb0c1d112719 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:21.406345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.407077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.408610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1280/2580] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:21 2022] DELETE /identity/v3/projects/f23f275251304aff8e92cb0c1d112719 => generated 0 bytes in 278 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:21.412973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:21.427047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf111b55-2e00-44a6-819e-1a2ce1d7b440 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:21.495229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:21.496304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:21.497208 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:21.497536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] PATH_INFO: `/v3/users/e1e4d248d53b42158e554d01ba586d9c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:21.498935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1e4d248d53b42158e554d01ba586d9c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:21.500808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:21.901169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1e4d248d53b42158e554d01ba586d9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.901169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1e4d248d53b42158e554d01ba586d9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.911668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1e4d248d53b42158e554d01ba586d9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.912411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1e4d248d53b42158e554d01ba586d9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.978237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-30929dba-b89c-433a-8661-784a63fa731b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1e4d248d53b42158e554d01ba586d9c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:21.998401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1301/2581] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:21 2022] DELETE /identity/v3/users/e1e4d248d53b42158e554d01ba586d9c => generated 0 bytes in 586 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:22.444260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:22.446432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:22.481337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a7f21c1-f802-456d-979f-2be7c35612a6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:22.561347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:22.562140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:22.562621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:22.563094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] PATH_INFO: `/v3/projects/9adda77e3aad43a78dbeb5c8d9c59178` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:22.564272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9adda77e3aad43a78dbeb5c8d9c59178)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:22.566185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:22.709045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9adda77e3aad43a78dbeb5c8d9c59178', 'request_id': 'req-5da9cc0d-9db0-43e7-a411-01a54f2a673c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:22.715114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9adda77e3aad43a78dbeb5c8d9c59178', 'request_id': 'req-5da9cc0d-9db0-43e7-a411-01a54f2a673c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:22.861436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9adda77e3aad43a78dbeb5c8d9c59178', 'request_id': 'req-5da9cc0d-9db0-43e7-a411-01a54f2a673c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:22.875047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] The token cache is being invalidate because project 9adda77e3aad43a78dbeb5c8d9c59178 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:22.875462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:22.876179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5da9cc0d-9db0-43e7-a411-01a54f2a673c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:22.877739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1281/2582] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:22 2022] DELETE /identity/v3/projects/9adda77e3aad43a78dbeb5c8d9c59178 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:22.882615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:22.883165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:22.883574 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:22.894345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:22.905867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:22.907135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:22.971932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1302/2583] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:44:22.980513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:22.981403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:22.981792 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.010760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:23.010760 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:23.025729 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1282/2584] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:22 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 46 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:23.026533 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.039820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61a7194-5243-4192-ba03-0ade744444e4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.047405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.047405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.047405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.047405 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.047838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:44:23.047838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.050292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.065356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1303/2585] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:44:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:44:23.069946 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.080130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74659ecd-08fd-4545-a03d-b74864553185 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.088555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.089418 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.089706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.090029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.090936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.092908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.190873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1283/2586] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:23 2022] POST /identity/v3/projects => generated 377 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:23.195857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.206729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-937e48b6-db4b-4851-81d4-2677c9aa7c9d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.273782 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.274303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.274572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.275427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.275563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.277764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.366697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1304/2587] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:23 2022] POST /identity/v3/users => generated 289 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:23.370704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.381165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3333b471-b021-4574-a47c-5919311274cb admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.385888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.386487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.386768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.387069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.388037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.392772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.400121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1284/2588] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:44:23.404969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.415045 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a1a5758-71b8-40c8-9602-85f964d407e6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.420284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1e4aac-2359-4428-b70a-49bd7b8cd993 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.421349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1e4aac-2359-4428-b70a-49bd7b8cd993 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.421349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1e4aac-2359-4428-b70a-49bd7b8cd993 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.421553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1e4aac-2359-4428-b70a-49bd7b8cd993 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.422014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1e4aac-2359-4428-b70a-49bd7b8cd993 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.423663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1e4aac-2359-4428-b70a-49bd7b8cd993 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.434456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1305/2589] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:23 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:23.438616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.454660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6851974-01df-425a-a225-d4aca26e88a0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.454660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.455333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.455597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.456108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] PATH_INFO: `/v3/projects/386c4e11620e46f1942f797b84a6746f/users/f3b9e0899b834fe59b83095fdcbfb7e1/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.459453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=85,keystone:DELETE=12,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:23.472245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] RBAC: Authorizing `identity:create_grant(project_id=386c4e11620e46f1942f797b84a6746f, user_id=f3b9e0899b834fe59b83095fdcbfb7e1, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.474647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.572420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1285/2590] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:23 2022] PUT /identity/v3/projects/386c4e11620e46f1942f797b84a6746f/users/f3b9e0899b834fe59b83095fdcbfb7e1/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:23.582665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.582981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.587861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.611415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] MFA Rules not processed for user `f3b9e0899b834fe59b83095fdcbfb7e1`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:23.619420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:23.676654 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:23.680283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1306/2591] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:23 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:23.686183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.750382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19d8b3f7-5b57-4eda-b43f-6b302e46a301 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.755130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.755741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.756151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.756259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.756889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.759224 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:23.909748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1286/2592] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:23 2022] POST /identity/v3/projects => generated 379 bytes in 225 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:23.913828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:23.925039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c9a87b-26d8-4513-9965-99bfe635edd7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:23.984648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:23.985696 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:23.986182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:23.986691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:23.987859 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:23.991252 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.086673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1307/2593] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:23 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:24.091330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.101862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7a64cbb-5577-4f61-abda-a18ff29ffacf admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.107728 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.108329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.108613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.108894 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.109577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.111252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.122171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1287/2594] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:24.126279 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.136572 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21711d9a-a945-45d9-9e6f-0105acd862fa admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.141500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3a3d950-b98a-47cc-b647-3831b370b36d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.142148 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3d950-b98a-47cc-b647-3831b370b36d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.142428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3d950-b98a-47cc-b647-3831b370b36d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.142699 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3a3d950-b98a-47cc-b647-3831b370b36d admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.143426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3d950-b98a-47cc-b647-3831b370b36d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.145003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3a3d950-b98a-47cc-b647-3831b370b36d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.156770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1308/2595] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:24 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:24.161238 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.172420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c485d47-0266-4d59-b46e-8199254a1cb9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.177269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.177949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.178222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.178487 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] PATH_INFO: `/v3/projects/1feabde906fb4392a928b9674adcc491/users/4e020b9e80414126b598e791d697a268/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.193110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1feabde906fb4392a928b9674adcc491, user_id=4e020b9e80414126b598e791d697a268, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.195811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.287007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1288/2596] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:24 2022] PUT /identity/v3/projects/1feabde906fb4392a928b9674adcc491/users/4e020b9e80414126b598e791d697a268/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:44:24.297840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.298121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.298390 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.447866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] MFA Rules not processed for user `4e020b9e80414126b598e791d697a268`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:24.449025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:24.512551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1309/2597] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:24 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 216 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:24.517676 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.588800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c7888cf-9361-4d56-91b2-5af45df15f15 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.589198 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.589916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.590076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.590361 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.591016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.593730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.681688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1289/2598] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:24 2022] POST /identity/v3/projects => generated 379 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:24.685874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.696516 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ac5b595-7750-454a-9693-5751738232c4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.751847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.752473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.752741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.753085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.753748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.755898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.850447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:24.856785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1310/2599] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:24 2022] POST /identity/v3/users => generated 296 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:24.860929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.871745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b6b799a-3fd3-4a4a-b08d-416d27babf97 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.877272 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.877872 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.878149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.878424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.879244 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.880842 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.892984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1290/2600] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:24.896561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.906999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfcc46dd-bdfc-47f2-9d33-34ab02eee25c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:24.912282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:24.912934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:24.913205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:24.913478 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] PATH_INFO: `/v3/projects/4aa2261a714648559413f54924a07c74/users/dd376c4073b6494ab8e640dcfb2b0c6f/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:24.928540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] RBAC: Authorizing `identity:create_grant(project_id=4aa2261a714648559413f54924a07c74, user_id=dd376c4073b6494ab8e640dcfb2b0c6f, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:24.931320 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:24.981171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1311/2601] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:24 2022] PUT /identity/v3/projects/4aa2261a714648559413f54924a07c74/users/dd376c4073b6494ab8e640dcfb2b0c6f/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 85 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:24.987440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:24.997743 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bdf4712-3f2f-4798-811c-0ae396e7fe18 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:25.002360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:25.055191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-445634ed-fb63-4125-9996-b6f3f0bc9c13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:25.055796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445634ed-fb63-4125-9996-b6f3f0bc9c13 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:25.056070 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445634ed-fb63-4125-9996-b6f3f0bc9c13 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:25.056423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-445634ed-fb63-4125-9996-b6f3f0bc9c13 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:25.057364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445634ed-fb63-4125-9996-b6f3f0bc9c13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:25.059047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-445634ed-fb63-4125-9996-b6f3f0bc9c13 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:25.069476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1291/2602] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:24 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:25.073866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:25.084640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f72d5247-fc59-4b6f-aecb-6edd2bd2d6ce admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:25.089354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:25.089990 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:25.090260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:25.090521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] PATH_INFO: `/v3/projects/4aa2261a714648559413f54924a07c74/users/dd376c4073b6494ab8e640dcfb2b0c6f/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:25.093003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4aa2261a714648559413f54924a07c74, user_id=dd376c4073b6494ab8e640dcfb2b0c6f, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:25.099130 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:25.105671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1312/2603] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:25 2022] PUT /identity/v3/projects/4aa2261a714648559413f54924a07c74/users/dd376c4073b6494ab8e640dcfb2b0c6f/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:25.117603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:25.118212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:25.118629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:25.147061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] MFA Rules not processed for user `dd376c4073b6494ab8e640dcfb2b0c6f`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:25.148262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:25.215298 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1292/2604] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:25 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:25.227329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:25.228979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:25.285612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2024e570-4bfa-45ef-b37a-76ba7fc6f7a0 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:25.289211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:25.289801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:25.290073 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:25.290341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:25.299205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:25.300730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:25.309897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1313/2605] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:25 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:25.408203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:25.409605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:25.421350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-107a298b-dc77-49b8-8512-07efe08217bf None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:25.424939 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b166ee0a-3011-43ca-88e6-44d39fc39f73 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:25.425534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166ee0a-3011-43ca-88e6-44d39fc39f73 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:25.425807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166ee0a-3011-43ca-88e6-44d39fc39f73 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:25.426075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b166ee0a-3011-43ca-88e6-44d39fc39f73 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:25.482297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b166ee0a-3011-43ca-88e6-44d39fc39f73 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:25.483966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b166ee0a-3011-43ca-88e6-44d39fc39f73 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:25.492998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1293/2606] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:25 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:29.897569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:29.960454 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cab8b56a-8ad3-4214-b601-931d128d581d service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:29.964956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:29.965565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:29.965839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:29.966113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] PATH_INFO: `/v3/users/f3b9e0899b834fe59b83095fdcbfb7e1` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:29.967285 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f3b9e0899b834fe59b83095fdcbfb7e1)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:29.969177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:30.209081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f3b9e0899b834fe59b83095fdcbfb7e1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:30.209757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f3b9e0899b834fe59b83095fdcbfb7e1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:30.216724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f3b9e0899b834fe59b83095fdcbfb7e1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:30.217839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f3b9e0899b834fe59b83095fdcbfb7e1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:30.312652 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f3b9e0899b834fe59b83095fdcbfb7e1'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:30.315053 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:INSERT=3,keystone:DELETE=8 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:30.331121 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1314/2607] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:29 2022] DELETE /identity/v3/users/f3b9e0899b834fe59b83095fdcbfb7e1 => generated 0 bytes in 434 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:30.335555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:30.336056 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:30.336155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:30.356376 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:30.357587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:30.383420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:30.418714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1294/2608] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:30 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:30.429075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:30.430492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:30.486916 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d06bc4d3-c7f7-4b2b-bc22-4caab0298fa5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:30.490388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:30.491010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:30.491284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:30.491548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:30.500987 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:30.502407 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:30.512009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1315/2609] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:30 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:44:30.938714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:30.940382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:30.954812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab995937-d081-47db-bf5c-1adbdf894819 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:30.959805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:30.960591 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:30.960934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:30.961275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] PATH_INFO: `/v3/projects/386c4e11620e46f1942f797b84a6746f` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:30.962577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=386c4e11620e46f1942f797b84a6746f)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:30.964889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:31.095848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '386c4e11620e46f1942f797b84a6746f', 'request_id': 'req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.101831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '386c4e11620e46f1942f797b84a6746f', 'request_id': 'req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.172623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '386c4e11620e46f1942f797b84a6746f', 'request_id': 'req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.187110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] The token cache is being invalidate because project 386c4e11620e46f1942f797b84a6746f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:31.187532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.188300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.189916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1295/2610] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:30 2022] DELETE /identity/v3/projects/386c4e11620e46f1942f797b84a6746f => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:31.194152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:31.204608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23ce85bd-0f8a-499c-a4a0-b4ee73b2dd2c service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:31.262457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:31.263112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:31.263389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:31.263660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] PATH_INFO: `/v3/users/4e020b9e80414126b598e791d697a268` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:31.265450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4e020b9e80414126b598e791d697a268)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:31.267398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:31.475589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4e020b9e80414126b598e791d697a268'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.476196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4e020b9e80414126b598e791d697a268'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.484005 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4e020b9e80414126b598e791d697a268'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.484752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4e020b9e80414126b598e791d697a268'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.611862 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4e020b9e80414126b598e791d697a268'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:31.631501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1316/2611] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:31 2022] DELETE /identity/v3/users/4e020b9e80414126b598e791d697a268 => generated 0 bytes in 438 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:32.249215 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:32.251770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:32.265521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85cb955c-e8a9-4b7f-883d-175ea3fd71c6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:32.330757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:32.331966 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:32.332603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:32.332879 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] PATH_INFO: `/v3/projects/1feabde906fb4392a928b9674adcc491` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:32.334025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1feabde906fb4392a928b9674adcc491)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:32.335973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:32.477151 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1feabde906fb4392a928b9674adcc491', 'request_id': 'req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.481607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:32.487624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1feabde906fb4392a928b9674adcc491', 'request_id': 'req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.589305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1feabde906fb4392a928b9674adcc491', 'request_id': 'req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.602166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] The token cache is being invalidate because project 1feabde906fb4392a928b9674adcc491 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:32.602463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.603148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.604740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1296/2612] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:32 2022] DELETE /identity/v3/projects/1feabde906fb4392a928b9674adcc491 => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:32.609467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:32.611088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:32.620316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:32.624509 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-907733b4-0548-4bb8-9db9-98d8debe1719 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:32.680311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:32.680935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:32.681230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:32.681497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] PATH_INFO: `/v3/users/dd376c4073b6494ab8e640dcfb2b0c6f` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:32.682619 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] RBAC: Authorizing `identity:delete_user(user_id=dd376c4073b6494ab8e640dcfb2b0c6f)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:32.684641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:32.954757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dd376c4073b6494ab8e640dcfb2b0c6f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.955430 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dd376c4073b6494ab8e640dcfb2b0c6f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.963593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dd376c4073b6494ab8e640dcfb2b0c6f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:32.964265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dd376c4073b6494ab8e640dcfb2b0c6f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.046482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ca534f74-36f5-437e-a1b0-b75b8362d670 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dd376c4073b6494ab8e640dcfb2b0c6f'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.072107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1317/2613] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:32 2022] DELETE /identity/v3/users/dd376c4073b6494ab8e640dcfb2b0c6f => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:33.619191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:33.619191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:33.633080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d37eb15e-7592-4fec-ad2e-b56a1de72dc2 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:33.705814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:33.706463 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:33.706748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:33.707083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] PATH_INFO: `/v3/projects/4aa2261a714648559413f54924a07c74` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:33.708261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4aa2261a714648559413f54924a07c74)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:33.711572 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:33.845116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4aa2261a714648559413f54924a07c74', 'request_id': 'req-8ffa2e51-899a-47f1-9293-387093a97d58'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.851095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4aa2261a714648559413f54924a07c74', 'request_id': 'req-8ffa2e51-899a-47f1-9293-387093a97d58'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.920127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4aa2261a714648559413f54924a07c74', 'request_id': 'req-8ffa2e51-899a-47f1-9293-387093a97d58'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.933057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] The token cache is being invalidate because project 4aa2261a714648559413f54924a07c74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:33.933354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.934000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8ffa2e51-899a-47f1-9293-387093a97d58 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:33.935586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1297/2614] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:33 2022] DELETE /identity/v3/projects/4aa2261a714648559413f54924a07c74 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:33.942060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:33.942563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:33.942673 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:33.961632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:33.962853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:34.021895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1318/2615] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:33 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:34.031188 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.031469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.031754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.053433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:34.057797 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:34.063278 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1298/2616] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:34 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:34.067600 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.077644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f78d0d-9f93-45ad-9fe6-6263b8433528 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.082561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.083193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.083466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.083729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.084368 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:44:34.084663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.086280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.098808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1319/2617] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:44:34 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:44:34.103594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.114486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b2b6a1-84a9-45b2-9330-1bd35cc9ad2b None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.119303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.119933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.120227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.120907 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.121125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.123053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.230525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1299/2618] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:34 2022] POST /identity/v3/projects => generated 369 bytes in 127 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:34.234747 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.244725 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cd20bf4-5029-4ad3-a861-902a22ae6101 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.253783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:34.306456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.307091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.307367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.307633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.308270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.310171 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.393309 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1320/2619] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:34 2022] POST /identity/v3/users => generated 285 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:44:34.397498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.409318 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96d568b4-61a5-4a9f-8ab6-df4960f4863c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.413982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.414586 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.414906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.415707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.415795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.417382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.427945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1300/2620] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:34 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:34.432057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.446200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-612310c5-88dd-4a53-b2d3-725c661d1820 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.452014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7da4c85f-f971-4232-ab02-3b6a76b1b4b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.452894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4c85f-f971-4232-ab02-3b6a76b1b4b1 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.453174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4c85f-f971-4232-ab02-3b6a76b1b4b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.454339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7da4c85f-f971-4232-ab02-3b6a76b1b4b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.454339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4c85f-f971-4232-ab02-3b6a76b1b4b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.455705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7da4c85f-f971-4232-ab02-3b6a76b1b4b1 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.466995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1321/2621] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:34 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:34.471873 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.482585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31cfe739-e114-4d52-bbb4-0eaf3abbaef3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.488932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.489610 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.492938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.492938 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] PATH_INFO: `/v3/projects/914529a2fedc4e699f037c1ef6d0ee40/users/6f40963732654c9fa66cbb85cd7fa0ea/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.506695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=914529a2fedc4e699f037c1ef6d0ee40, user_id=6f40963732654c9fa66cbb85cd7fa0ea, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.509408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.575841 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1301/2622] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:34 2022] PUT /identity/v3/projects/914529a2fedc4e699f037c1ef6d0ee40/users/6f40963732654c9fa66cbb85cd7fa0ea/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:34.588838 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.589117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.589391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.616736 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] MFA Rules not processed for user `6f40963732654c9fa66cbb85cd7fa0ea`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:34.617952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:34.690653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1322/2623] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:34 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:34.702953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:34.708031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:34.715758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=9,keystone:SELECT=87,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:34.762993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-009a0f07-2df2-4e46-8cf1-0607f6f1d3b9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:34.767585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3855cb61-d3a1-4653-9d7c-65c1aae93bd4 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '6a27d7ca876a4320bb05f2afc86bee3d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:34.768217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855cb61-d3a1-4653-9d7c-65c1aae93bd4 service glance] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:34.768632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855cb61-d3a1-4653-9d7c-65c1aae93bd4 service glance] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:34.768911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3855cb61-d3a1-4653-9d7c-65c1aae93bd4 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:34.777762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855cb61-d3a1-4653-9d7c-65c1aae93bd4 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:34.779228 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3855cb61-d3a1-4653-9d7c-65c1aae93bd4 service glance] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:34.788601 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1302/2624] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:34 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:37.050858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:37.052251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:37.124732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a01f32c-d7ab-4b98-88dd-fb068590049b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:37.129164 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:37.129784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:37.130061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:37.130330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] PATH_INFO: `/v3/users/c8b735eb05fa417880f159c384b0b4d7` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:37.131457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c8b735eb05fa417880f159c384b0b4d7)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:37.133367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:37.154756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=94,keystone:INSERT=3,keystone:DELETE=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:37.331595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c8b735eb05fa417880f159c384b0b4d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:37.332264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c8b735eb05fa417880f159c384b0b4d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:37.339911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c8b735eb05fa417880f159c384b0b4d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:37.340668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c8b735eb05fa417880f159c384b0b4d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:37.432668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c8b735eb05fa417880f159c384b0b4d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:37.451917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1323/2625] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:37 2022] DELETE /identity/v3/users/c8b735eb05fa417880f159c384b0b4d7 => generated 0 bytes in 402 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:44:37.456573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:37.457037 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:37.457134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:37.476495 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:37.477702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:37.542342 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1303/2626] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:37 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:37.550962 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:37.552385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:37.609181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e24d1f60-4322-403f-8a24-e90e3d5db694 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:37.612766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:37.613369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:37.613641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:37.613905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:37.624683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:37.626217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:37.636887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1324/2627] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:38.063471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:38.064920 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:38.078333 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79077557-40bd-4e79-a956-07d2bb20fee1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:38.082987 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:38.083618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:38.083891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:38.084301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] PATH_INFO: `/v3/projects/de0c370521644f77b1abcc3c0acdb7ce` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:38.085345 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=de0c370521644f77b1abcc3c0acdb7ce)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:38.087276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:38.218609 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'de0c370521644f77b1abcc3c0acdb7ce', 'request_id': 'req-ea25e352-8f78-4692-a27f-252b7c6def31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.224427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'de0c370521644f77b1abcc3c0acdb7ce', 'request_id': 'req-ea25e352-8f78-4692-a27f-252b7c6def31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.294983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'de0c370521644f77b1abcc3c0acdb7ce', 'request_id': 'req-ea25e352-8f78-4692-a27f-252b7c6def31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.311554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] The token cache is being invalidate because project de0c370521644f77b1abcc3c0acdb7ce was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:38.311861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.312682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.314702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1304/2628] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:38 2022] DELETE /identity/v3/projects/de0c370521644f77b1abcc3c0acdb7ce => generated 0 bytes in 252 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:38.319346 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:38.333457 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b54bec7-1b0d-426e-b112-c0cd567aa270 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:38.401060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:38.401697 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:38.402224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:38.402511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] PATH_INFO: `/v3/users/b008fb898ec54c0d99b7f078a8b38d6d` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:38.403640 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b008fb898ec54c0d99b7f078a8b38d6d)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:38.405642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:38.630443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=13,keystone:SELECT=86,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:38.632839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b008fb898ec54c0d99b7f078a8b38d6d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.633750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b008fb898ec54c0d99b7f078a8b38d6d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.640330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b008fb898ec54c0d99b7f078a8b38d6d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.640972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b008fb898ec54c0d99b7f078a8b38d6d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.725610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-70b647d9-16e2-4c68-ad74-bdf6a38fa3d2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b008fb898ec54c0d99b7f078a8b38d6d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:38.745039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1325/2629] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:38 2022] DELETE /identity/v3/users/b008fb898ec54c0d99b7f078a8b38d6d => generated 0 bytes in 426 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:39.154325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:39.154325 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:39.167358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea25e352-8f78-4692-a27f-252b7c6def31 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:39.202338 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:39.227727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:39.228488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.228770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.229195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] PATH_INFO: `/v3/projects/c30761d86cb748b9a1b02e3cad81b8bc` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.230427 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c30761d86cb748b9a1b02e3cad81b8bc)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:39.233847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:39.367882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c30761d86cb748b9a1b02e3cad81b8bc', 'request_id': 'req-eea3c1a1-0c32-4487-bb6d-6bd027167117'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:39.373704 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c30761d86cb748b9a1b02e3cad81b8bc', 'request_id': 'req-eea3c1a1-0c32-4487-bb6d-6bd027167117'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:39.443829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c30761d86cb748b9a1b02e3cad81b8bc', 'request_id': 'req-eea3c1a1-0c32-4487-bb6d-6bd027167117'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:39.456204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] The token cache is being invalidate because project c30761d86cb748b9a1b02e3cad81b8bc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:39.456488 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:39.457121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eea3c1a1-0c32-4487-bb6d-6bd027167117 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:39.458639 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1305/2630] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:39 2022] DELETE /identity/v3/projects/c30761d86cb748b9a1b02e3cad81b8bc => generated 0 bytes in 310 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:39.463773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.464296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.464472 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.485131 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:39.486703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:39.549558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1326/2631] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:44:39.557504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.557789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.558090 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.578261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:39.579498 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:39.587499 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1306/2632] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:39.591808 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:39.602985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c836d887-64a2-4c6c-921b-7208ad088fe3 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:39.608566 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:39.609818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.609818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.610041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.610853 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:44:39.611027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:39.612695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:39.625972 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1327/2633] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:44:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:44:39.630540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:39.640293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-201eb693-2e99-4a73-9f47-1dc9101e5ea1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:39.645834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:39.646432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.646708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.647008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.647641 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:39.649585 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:39.779147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1307/2634] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:39 2022] POST /identity/v3/projects => generated 379 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:44:39.785378 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:39.797462 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd1ab4a4-fcb7-46b5-a5dd-7cddb9b3b88f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:39.855878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:39.856475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.856745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.857072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.857846 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:39.859765 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:39.973729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1328/2635] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:39 2022] POST /identity/v3/users => generated 290 bytes in 190 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:44:39.978166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:39.988963 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f10b0563-e635-4c32-907e-a3fc90c2bc88 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:39.993615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:39.994208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:39.994501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:39.994753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:39.995412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:39.996985 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.006977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1308/2636] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:39 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:40.013310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.023677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82219153-8fb2-45cc-b2a2-5753caac2d14 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.028913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87903380-ffb8-4b82-9ab2-df2cf910559e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.030361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87903380-ffb8-4b82-9ab2-df2cf910559e admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.030948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87903380-ffb8-4b82-9ab2-df2cf910559e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.031142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87903380-ffb8-4b82-9ab2-df2cf910559e admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.031830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87903380-ffb8-4b82-9ab2-df2cf910559e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.033422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87903380-ffb8-4b82-9ab2-df2cf910559e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.044297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1329/2637] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:40.049261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.059942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecf739f2-8d51-456e-b647-041939cb1416 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.064802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.065673 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.065968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.066435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] PATH_INFO: `/v3/projects/e3cc06c1f3844a9894af6e162cc706f1/users/b67ab2ead9c4424a86ea85e6b0fbe93a/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.082362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] RBAC: Authorizing `identity:create_grant(project_id=e3cc06c1f3844a9894af6e162cc706f1, user_id=b67ab2ead9c4424a86ea85e6b0fbe93a, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.085100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.161241 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1309/2638] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:40 2022] PUT /identity/v3/projects/e3cc06c1f3844a9894af6e162cc706f1/users/b67ab2ead9c4424a86ea85e6b0fbe93a/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:40.172653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.172938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.173295 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.200331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] MFA Rules not processed for user `b67ab2ead9c4424a86ea85e6b0fbe93a`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:40.202165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:40.214539 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=4,keystone:SELECT=92,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:40.267513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1330/2639] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:40 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:44:40.272838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.338412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d899696-a3de-432d-b9fa-a24b1b25501d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.343882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.344557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.344789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.345130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.345774 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.347735 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.353323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:40.478479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1310/2640] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:40 2022] POST /identity/v3/projects => generated 381 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:40.483735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.494651 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de694a2c-95ad-4756-bcbf-45d337bcdded None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.551796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.552395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.552804 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.552926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.553551 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.555554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.638559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1331/2641] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:40 2022] POST /identity/v3/users => generated 297 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:40.643309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.656806 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8051f30d-ab64-4ee4-b849-5063343c8c37 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.662537 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.663343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.663632 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.663906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.664670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.666470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.680101 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1311/2642] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:40 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:40.684161 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.694949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dc63d5f-cacb-47b8-9885-a712fd30df9c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.699569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.700224 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.700496 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.700762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] PATH_INFO: `/v3/projects/adc629d993f64d1ca48e48c266962ce4/users/a249aff1e1074e7f9dbc8956becea987/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.715841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] RBAC: Authorizing `identity:create_grant(project_id=adc629d993f64d1ca48e48c266962ce4, user_id=a249aff1e1074e7f9dbc8956becea987, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.718883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.823165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1332/2643] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:40 2022] PUT /identity/v3/projects/adc629d993f64d1ca48e48c266962ce4/users/a249aff1e1074e7f9dbc8956becea987/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:44:40.827229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.837523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4191944-91de-4436-b319-7a894b12b551 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.899256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e8065e7-1187-406a-a2f7-fa15c475b67f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.899865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8065e7-1187-406a-a2f7-fa15c475b67f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.900190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8065e7-1187-406a-a2f7-fa15c475b67f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.900470 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e8065e7-1187-406a-a2f7-fa15c475b67f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.901106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8065e7-1187-406a-a2f7-fa15c475b67f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.902990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e8065e7-1187-406a-a2f7-fa15c475b67f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.913504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1312/2644] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:40 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:40.917776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:40.928170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-971ba6fc-a608-4bbb-8458-313e35bcb17f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:40.932712 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:40.933356 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.933632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.933896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] PATH_INFO: `/v3/projects/adc629d993f64d1ca48e48c266962ce4/users/a249aff1e1074e7f9dbc8956becea987/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.935938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] RBAC: Authorizing `identity:create_grant(project_id=adc629d993f64d1ca48e48c266962ce4, user_id=a249aff1e1074e7f9dbc8956becea987, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:40.938650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:40.952832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1333/2645] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:40 2022] PUT /identity/v3/projects/adc629d993f64d1ca48e48c266962ce4/users/a249aff1e1074e7f9dbc8956becea987/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 36 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:40.963148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:40.963432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:40.963709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:40.991807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] MFA Rules not processed for user `a249aff1e1074e7f9dbc8956becea987`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:40.993017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:41.064667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1313/2646] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:40 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:41.075890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:41.078280 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:41.134977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66a4be61-8929-4326-a4dd-9e35ffa76a8e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:41.139926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:41.140550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:41.140831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:41.141099 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:41.150902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:41.152825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:41.162874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1334/2647] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:41 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:41.380161 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:41.390524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a23e504d-d15b-461f-aeda-1195343b0392 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:41.394222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb4b3c75-eaec-44c0-9166-da6245ac7f72 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:41.394867 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4b3c75-eaec-44c0-9166-da6245ac7f72 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:41.395142 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4b3c75-eaec-44c0-9166-da6245ac7f72 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:41.395411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb4b3c75-eaec-44c0-9166-da6245ac7f72 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:41.457805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:41.460128 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4b3c75-eaec-44c0-9166-da6245ac7f72 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:41.461624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb4b3c75-eaec-44c0-9166-da6245ac7f72 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:41.470514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1314/2648] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:41 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:51.158684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:51.519502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:53.113092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:53.175276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30089d7d-fd29-4018-aada-7af21a1ec5a0 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:53.179769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:53.180391 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:53.180667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:53.180934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] PATH_INFO: `/v3/users/b67ab2ead9c4424a86ea85e6b0fbe93a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:53.182027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b67ab2ead9c4424a86ea85e6b0fbe93a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:53.184044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:53.321380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b67ab2ead9c4424a86ea85e6b0fbe93a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:53.322027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b67ab2ead9c4424a86ea85e6b0fbe93a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:53.329596 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b67ab2ead9c4424a86ea85e6b0fbe93a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:53.330245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b67ab2ead9c4424a86ea85e6b0fbe93a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:53.382397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b67ab2ead9c4424a86ea85e6b0fbe93a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:53.400417 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1335/2649] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:53 2022] DELETE /identity/v3/users/b67ab2ead9c4424a86ea85e6b0fbe93a => generated 0 bytes in 288 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:53.405262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:53.405741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:53.405947 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:53.425302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:53.426574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:53.484012 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1315/2650] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:53 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:53.493755 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:53.495236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:53.553687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccbab6b4-94a6-4e96-bd61-7771cd1be2d5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:53.557773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:53.558374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:53.558648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:53.558949 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:53.570655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:53.572101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:53.581521 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1336/2651] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:53 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:54.001336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:54.002786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:54.015232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fb63d8-6db5-4e82-9f94-bb07d7f7c897 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:54.020614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:54.021247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:54.021518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:54.021831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] PATH_INFO: `/v3/projects/e3cc06c1f3844a9894af6e162cc706f1` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:54.022952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] RBAC: Authorizing `identity:delete_project(project_id=e3cc06c1f3844a9894af6e162cc706f1)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:54.024954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:54.188106 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e3cc06c1f3844a9894af6e162cc706f1', 'request_id': 'req-99847ec6-e870-4f85-b3d6-58c9211d3509'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.193772 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e3cc06c1f3844a9894af6e162cc706f1', 'request_id': 'req-99847ec6-e870-4f85-b3d6-58c9211d3509'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.244811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e3cc06c1f3844a9894af6e162cc706f1', 'request_id': 'req-99847ec6-e870-4f85-b3d6-58c9211d3509'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.258583 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] The token cache is being invalidate because project e3cc06c1f3844a9894af6e162cc706f1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:54.259109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.259778 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.261561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1316/2652] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:54 2022] DELETE /identity/v3/projects/e3cc06c1f3844a9894af6e162cc706f1 => generated 0 bytes in 261 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:54.265425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:54.275359 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8693cb8c-0da5-4034-a270-a56f1397b188 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:54.295147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:54.333193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:54.333815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:54.334091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:54.334387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] PATH_INFO: `/v3/users/a249aff1e1074e7f9dbc8956becea987` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:54.335590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] RBAC: Authorizing `identity:delete_user(user_id=a249aff1e1074e7f9dbc8956becea987)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:54.337506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:54.590209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a249aff1e1074e7f9dbc8956becea987'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.590795 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a249aff1e1074e7f9dbc8956becea987'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.598172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a249aff1e1074e7f9dbc8956becea987'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.598802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a249aff1e1074e7f9dbc8956becea987'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.684996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-cf54ba0b-b2d7-4416-a6f6-eb1d898e880a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a249aff1e1074e7f9dbc8956becea987'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:54.704991 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1337/2653] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:54 2022] DELETE /identity/v3/users/a249aff1e1074e7f9dbc8956becea987 => generated 0 bytes in 440 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:55.075615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.077097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:55.090360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99847ec6-e870-4f85-b3d6-58c9211d3509 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.146627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.147284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.147563 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.147832 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] PATH_INFO: `/v3/projects/adc629d993f64d1ca48e48c266962ce4` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.148952 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] RBAC: Authorizing `identity:delete_project(project_id=adc629d993f64d1ca48e48c266962ce4)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.150881 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:55.234122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:55.253096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'adc629d993f64d1ca48e48c266962ce4', 'request_id': 'req-b5e149f9-ca29-40a8-a926-1f22b5a0e821'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:55.258620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'adc629d993f64d1ca48e48c266962ce4', 'request_id': 'req-b5e149f9-ca29-40a8-a926-1f22b5a0e821'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:55.370739 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'adc629d993f64d1ca48e48c266962ce4', 'request_id': 'req-b5e149f9-ca29-40a8-a926-1f22b5a0e821'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:55.383160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] The token cache is being invalidate because project adc629d993f64d1ca48e48c266962ce4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:55.383452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:55.384083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-b5e149f9-ca29-40a8-a926-1f22b5a0e821 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:55.385579 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1317/2654] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:55 2022] DELETE /identity/v3/projects/adc629d993f64d1ca48e48c266962ce4 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:55.390503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.391081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.391367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.410703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:55.411905 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:55.472245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1338/2655] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:55.480949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.481232 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.481502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.501358 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:55.502538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:55.510307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1318/2656] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:55 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:55.514413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.524565 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deaa35cd-b802-4971-8e7f-47bda1d6cab1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.529426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.530013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.530292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.530562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.531240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:44:55.533108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.534756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:55.546169 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1339/2657] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:44:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:44:55.551017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.560083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e172f25-f98d-424a-bf0a-db327a1cb693 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.564959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.565540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.565815 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.566099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.566723 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.568629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:55.690534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1319/2658] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:55 2022] POST /identity/v3/projects => generated 373 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:55.694625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.704770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80dc251c-f171-40ec-b57a-8302c0fc5e9d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.730545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:55.760800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.761400 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.761678 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.761947 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.762573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.764511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:55.839392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1340/2659] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:55 2022] POST /identity/v3/users => generated 287 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:55.843566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.855045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87c13c32-cf5e-4d3e-924f-de6bc555fc75 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.860185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.860788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.861069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.861340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.862114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.863759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:55.873611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1320/2660] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:55.877819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.887649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c4774a8-9e22-4c07-9167-7968fd35642d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.892681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e1ab02d-f6ed-4be5-83da-7e138a5c8761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.893258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1ab02d-f6ed-4be5-83da-7e138a5c8761 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.893531 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1ab02d-f6ed-4be5-83da-7e138a5c8761 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.893796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e1ab02d-f6ed-4be5-83da-7e138a5c8761 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.894426 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1ab02d-f6ed-4be5-83da-7e138a5c8761 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.896031 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e1ab02d-f6ed-4be5-83da-7e138a5c8761 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:55.907119 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1341/2661] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:55 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:55.911284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:55.921349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03abeaa5-1074-4597-a349-f588ca73827c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:55.926015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:55.926662 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:55.926982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:55.927257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] PATH_INFO: `/v3/projects/391947198b1445f0acbf1f946fe06809/users/b692d70233ba4f95a35ee6c3b857f197/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:55.942811 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] RBAC: Authorizing `identity:create_grant(project_id=391947198b1445f0acbf1f946fe06809, user_id=b692d70233ba4f95a35ee6c3b857f197, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:55.945409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.004934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1321/2662] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:55 2022] PUT /identity/v3/projects/391947198b1445f0acbf1f946fe06809/users/b692d70233ba4f95a35ee6c3b857f197/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:56.015455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.016352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.016627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.045113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] MFA Rules not processed for user `b692d70233ba4f95a35ee6c3b857f197`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:56.046350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:56.110034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1342/2663] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:56 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:56.114788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.177116 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61c3aa0f-e153-4b32-9e90-500bf9458cac admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.181646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.182246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.182516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.182833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.183449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.185364 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.281292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1322/2664] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:44:56 2022] POST /identity/v3/projects => generated 373 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:56.285691 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.297269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c406699-1eda-45c9-9373-ba0bfa64b2af None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.360037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.360605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.360883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.361155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.361791 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.363763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.427866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1343/2665] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:44:56 2022] POST /identity/v3/users => generated 293 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:44:56.432159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.443337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b7c0d6-0576-49d8-a0ef-1ad7b5e7b0ad admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.447926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.448519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.448795 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.449102 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.449699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.451316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.464261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1323/2666] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:56 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:44:56.468480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.478386 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fa19434-65f3-436d-ac27-a55fbfec9538 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.482936 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.483564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.483839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.484106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] PATH_INFO: `/v3/projects/94986ec0c9a347f19e5e9f140c5b520a/users/e825ef9a14c14f94b8acdf73222308be/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.499409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94986ec0c9a347f19e5e9f140c5b520a, user_id=e825ef9a14c14f94b8acdf73222308be, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.502609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.590096 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1344/2667] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:56 2022] PUT /identity/v3/projects/94986ec0c9a347f19e5e9f140c5b520a/users/e825ef9a14c14f94b8acdf73222308be/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:56.595017 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.604892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e930d8-6fc3-42b1-bee5-b789e3f62216 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.624876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:56.661648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f19169b-9578-449e-951c-54289026ddb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.662258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f19169b-9578-449e-951c-54289026ddb2 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.662738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f19169b-9578-449e-951c-54289026ddb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.663407 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f19169b-9578-449e-951c-54289026ddb2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.664134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f19169b-9578-449e-951c-54289026ddb2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.665766 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f19169b-9578-449e-951c-54289026ddb2 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.675932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1324/2668] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:44:56 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:44:56.680239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.685575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:56.691648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1859302d-82f4-43ba-ae9c-4eb7583a2e75 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.696249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.696897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.697150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.697412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] PATH_INFO: `/v3/projects/94986ec0c9a347f19e5e9f140c5b520a/users/e825ef9a14c14f94b8acdf73222308be/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.699311 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] RBAC: Authorizing `identity:create_grant(project_id=94986ec0c9a347f19e5e9f140c5b520a, user_id=e825ef9a14c14f94b8acdf73222308be, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.701977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.712255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1345/2669] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:44:56 2022] PUT /identity/v3/projects/94986ec0c9a347f19e5e9f140c5b520a/users/e825ef9a14c14f94b8acdf73222308be/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:56.722517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07617734-a505-4cdf-8549-222b4f599981 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.722836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07617734-a505-4cdf-8549-222b4f599981 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.723115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07617734-a505-4cdf-8549-222b4f599981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.751956 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-07617734-a505-4cdf-8549-222b4f599981 None None] MFA Rules not processed for user `e825ef9a14c14f94b8acdf73222308be`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:56.753236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-07617734-a505-4cdf-8549-222b4f599981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:56.820097 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1325/2670] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:56 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:56.831344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:56.833009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:56.891337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-573ee90d-66ac-4a13-b2ee-ad2d71316310 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:56.896482 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-789d664f-3906-4576-a04f-67bcf44d98be service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:56.897088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d664f-3906-4576-a04f-67bcf44d98be service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:56.897362 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d664f-3906-4576-a04f-67bcf44d98be service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:56.897624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-789d664f-3906-4576-a04f-67bcf44d98be service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:56.909418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d664f-3906-4576-a04f-67bcf44d98be service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:56.910934 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-789d664f-3906-4576-a04f-67bcf44d98be service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:56.920786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1346/2671] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:56 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:58.010098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07617734-a505-4cdf-8549-222b4f599981 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:58.072901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07617734-a505-4cdf-8549-222b4f599981 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:58.077372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:58.077994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:58.078268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:58.078536 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] PATH_INFO: `/v3/users/b692d70233ba4f95a35ee6c3b857f197` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:58.079758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b692d70233ba4f95a35ee6c3b857f197)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:58.081666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:58.235898 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b692d70233ba4f95a35ee6c3b857f197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:58.236994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:58.242822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b692d70233ba4f95a35ee6c3b857f197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:58.243661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b692d70233ba4f95a35ee6c3b857f197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:58.244267 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b692d70233ba4f95a35ee6c3b857f197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:58.301828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b692d70233ba4f95a35ee6c3b857f197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:58.320754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1326/2672] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:58 2022] DELETE /identity/v3/users/b692d70233ba4f95a35ee6c3b857f197 => generated 0 bytes in 311 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:44:58.325278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:58.325849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:58.325849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:58.345888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:44:58.347145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:58.410120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1347/2673] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:44:58 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:44:58.419821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:58.421363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:58.478908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6680748-0e17-4f6a-ad52-b0dfdbd377e4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:58.482491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:58.483119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:58.483392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:58.483659 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:58.495165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:58.496649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:58.506027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1327/2674] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:44:58 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:44:58.988363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:58.989185 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:44:59.001929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620bc8b6-f7a4-456d-96c6-6185bfef4712 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:59.006680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:59.007401 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:59.007726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:59.007939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] PATH_INFO: `/v3/projects/391947198b1445f0acbf1f946fe06809` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:59.009046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] RBAC: Authorizing `identity:delete_project(project_id=391947198b1445f0acbf1f946fe06809)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:59.010998 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:59.114741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '391947198b1445f0acbf1f946fe06809', 'request_id': 'req-7df144e7-8e88-4b71-b1c0-87ce35a7e730'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.206138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '391947198b1445f0acbf1f946fe06809', 'request_id': 'req-7df144e7-8e88-4b71-b1c0-87ce35a7e730'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.213025 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '391947198b1445f0acbf1f946fe06809', 'request_id': 'req-7df144e7-8e88-4b71-b1c0-87ce35a7e730'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.223740 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] The token cache is being invalidate because project 391947198b1445f0acbf1f946fe06809 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:44:59.224030 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.224663 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.226258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1348/2675] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:44:58 2022] DELETE /identity/v3/projects/391947198b1445f0acbf1f946fe06809 => generated 0 bytes in 239 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:44:59.230321 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:44:59.240149 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c9aca69-69d9-46df-9609-2cfa6ac2b633 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:44:59.297791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:44:59.298439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:44:59.298717 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:44:59.299193 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] PATH_INFO: `/v3/users/e825ef9a14c14f94b8acdf73222308be` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:44:59.300330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e825ef9a14c14f94b8acdf73222308be)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:44:59.302264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:44:59.482569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e825ef9a14c14f94b8acdf73222308be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.489665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e825ef9a14c14f94b8acdf73222308be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.490359 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e825ef9a14c14f94b8acdf73222308be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.490980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e825ef9a14c14f94b8acdf73222308be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.570616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-d3f19c8d-d997-46d1-b76d-a87628873d04 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e825ef9a14c14f94b8acdf73222308be'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:44:59.576658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=86,keystone:INSERT=2,keystone:DELETE=12 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:44:59.589381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1328/2676] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:44:59 2022] DELETE /identity/v3/users/e825ef9a14c14f94b8acdf73222308be => generated 0 bytes in 360 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:00.012329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.013910 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:00.027661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7df144e7-8e88-4b71-b1c0-87ce35a7e730 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.039973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:00.083146 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.083708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.083980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.084249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] PATH_INFO: `/v3/projects/94986ec0c9a347f19e5e9f140c5b520a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.085348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=94986ec0c9a347f19e5e9f140c5b520a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.087282 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.210315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '94986ec0c9a347f19e5e9f140c5b520a', 'request_id': 'req-2d301001-e54a-49aa-a818-e2d52c6384d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:00.313951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '94986ec0c9a347f19e5e9f140c5b520a', 'request_id': 'req-2d301001-e54a-49aa-a818-e2d52c6384d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:00.320915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '94986ec0c9a347f19e5e9f140c5b520a', 'request_id': 'req-2d301001-e54a-49aa-a818-e2d52c6384d7'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:00.332203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] The token cache is being invalidate because project 94986ec0c9a347f19e5e9f140c5b520a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:45:00.332495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:00.333145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-2d301001-e54a-49aa-a818-e2d52c6384d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:00.334684 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1349/2677] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:45:00 2022] DELETE /identity/v3/projects/94986ec0c9a347f19e5e9f140c5b520a => generated 0 bytes in 323 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:00.339715 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.341274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.341274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.363411 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:00.364672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:00.425003 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1329/2678] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:00.436578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.436895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.437175 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.458418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:00.459660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:00.467573 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1350/2679] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:00.472748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.482240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58cb5414-06d7-440f-bf28-cffbcf2c41a0 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.486863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.487451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.487727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.487994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.488630 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:45:00.488929 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.490511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.501949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1330/2680] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:45:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:45:00.506209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.515439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62fd8def-9a40-432b-9dd1-c0c2588f89e1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.521796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.522447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.522726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.523065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.523698 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.525660 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.610608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1351/2681] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:45:00 2022] POST /identity/v3/projects => generated 367 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:00.614823 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.624961 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfd500a-a9a5-4769-bf00-41403beb8d00 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.681820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.682421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.682699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.682997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.683748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.685587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.781243 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1331/2682] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:45:00 2022] POST /identity/v3/users => generated 284 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:00.785645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.795822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912de60d-c2e9-4605-93d1-a86a47762821 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.800928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.801543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.801816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.802380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.803058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.804664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.815167 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1352/2683] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:00 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:00.819398 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.831270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aa090f1-1379-49e8-82ba-5e0fc1020a75 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.836115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e0d1bfc-0742-44c2-aa21-997d304a7649 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.836713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d1bfc-0742-44c2-aa21-997d304a7649 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.837045 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d1bfc-0742-44c2-aa21-997d304a7649 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.837317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e0d1bfc-0742-44c2-aa21-997d304a7649 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.837983 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0d1bfc-0742-44c2-aa21-997d304a7649 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.839616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e0d1bfc-0742-44c2-aa21-997d304a7649 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.850439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1332/2684] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:00.854723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:00.864874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8e85e4e-eac6-495c-8e0b-ce1259118aad admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:00.869641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:00.870274 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.870548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.870843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] PATH_INFO: `/v3/projects/ab48af86528c45a784554040c455be58/users/3d95b09ef2f243c794152bfc2c0b7efa/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:00.886114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] RBAC: Authorizing `identity:create_grant(project_id=ab48af86528c45a784554040c455be58, user_id=3d95b09ef2f243c794152bfc2c0b7efa, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:00.888770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:00.981648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1353/2685] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:00 2022] PUT /identity/v3/projects/ab48af86528c45a784554040c455be58/users/3d95b09ef2f243c794152bfc2c0b7efa/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:45:00.991875 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:00.994474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:00.994474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.020631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] MFA Rules not processed for user `3d95b09ef2f243c794152bfc2c0b7efa`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:01.021834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:01.063047 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:01.085834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1333/2686] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:00 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:01.090481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.134919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:01.153375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36514b4f-3b35-4374-8483-e814848bda9f admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.158688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.159314 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.159588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.159850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.160466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.162403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.260609 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1354/2687] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:45:01 2022] POST /identity/v3/projects => generated 367 bytes in 167 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:01.262177 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.273095 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb68d8df-3ec7-46c1-92bd-91c8608969f9 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.331324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.332093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.332367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.332637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.333275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.335350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.422108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1334/2688] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:45:01 2022] POST /identity/v3/users => generated 290 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:01.427084 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.437233 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff8cd68-2849-4307-b302-42fac9b88d9a admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.441902 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.442487 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.442758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.443060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.443686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.445281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.455370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1355/2689] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:01 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:01.459619 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.469871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0855592c-3b4a-45b7-9ba3-a8d54a1140ab admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.475422 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.476080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.476363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.476642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] PATH_INFO: `/v3/projects/581ae10f09bb4800878d721255a3f624/users/604605e987cd4453a3d9834bf46de8fa/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.492366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] RBAC: Authorizing `identity:create_grant(project_id=581ae10f09bb4800878d721255a3f624, user_id=604605e987cd4453a3d9834bf46de8fa, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.495172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.582382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1335/2690] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:01 2022] PUT /identity/v3/projects/581ae10f09bb4800878d721255a3f624/users/604605e987cd4453a3d9834bf46de8fa/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:01.586333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.597237 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-194c7e5a-6500-4293-93b1-abed1c40fd37 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.656824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49af20e6-977b-4a28-9954-7699fc42e3f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.658059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49af20e6-977b-4a28-9954-7699fc42e3f0 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.658340 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49af20e6-977b-4a28-9954-7699fc42e3f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.658730 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49af20e6-977b-4a28-9954-7699fc42e3f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.659425 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49af20e6-977b-4a28-9954-7699fc42e3f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.661141 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49af20e6-977b-4a28-9954-7699fc42e3f0 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.671771 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1356/2691] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:01 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:01.676650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.687120 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d015bdae-ec42-4613-87c7-2340559c684f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.691986 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.692637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.692917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.693204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] PATH_INFO: `/v3/projects/581ae10f09bb4800878d721255a3f624/users/604605e987cd4453a3d9834bf46de8fa/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.695147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] RBAC: Authorizing `identity:create_grant(project_id=581ae10f09bb4800878d721255a3f624, user_id=604605e987cd4453a3d9834bf46de8fa, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.697807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.709392 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1336/2692] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:01 2022] PUT /identity/v3/projects/581ae10f09bb4800878d721255a3f624/users/604605e987cd4453a3d9834bf46de8fa/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:01.721350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.721644 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.721915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.768709 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] MFA Rules not processed for user `604605e987cd4453a3d9834bf46de8fa`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:01.769971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:01.842583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1357/2693] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:01 2022] POST /identity/v3/auth/tokens => generated 3311 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:01.854744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:01.857448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:01.916928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-506adb8d-b16d-4987-973c-2092dd1ab65e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:01.920671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-479a2754-8708-4935-b88b-4ddd4a22368f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:01.921276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479a2754-8708-4935-b88b-4ddd4a22368f service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:01.921557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479a2754-8708-4935-b88b-4ddd4a22368f service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:01.921831 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-479a2754-8708-4935-b88b-4ddd4a22368f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:01.932594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479a2754-8708-4935-b88b-4ddd4a22368f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:01.932594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-479a2754-8708-4935-b88b-4ddd4a22368f service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:01.941865 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1337/2694] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:01 2022] GET /identity/v3/auth/tokens => generated 3311 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:04.707013 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:04.747822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:04.769061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cefc1666-623c-4230-a9db-ff60d9fe538a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:04.775577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:04.775577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:04.775577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:04.775577 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] PATH_INFO: `/v3/users/3d95b09ef2f243c794152bfc2c0b7efa` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:04.776052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] RBAC: Authorizing `identity:delete_user(user_id=3d95b09ef2f243c794152bfc2c0b7efa)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:04.777989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:04.926000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '3d95b09ef2f243c794152bfc2c0b7efa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:04.926646 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '3d95b09ef2f243c794152bfc2c0b7efa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:04.933735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '3d95b09ef2f243c794152bfc2c0b7efa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:04.934419 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '3d95b09ef2f243c794152bfc2c0b7efa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:04.981969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '3d95b09ef2f243c794152bfc2c0b7efa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:04.999394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1358/2695] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:45:04 2022] DELETE /identity/v3/users/3d95b09ef2f243c794152bfc2c0b7efa => generated 0 bytes in 293 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:05.004051 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:05.004517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:05.004631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:05.024028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:05.025218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:05.058141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:05.085783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1338/2696] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:05 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:05.094683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:05.098465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:05.152966 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574102bd-cdff-4941-a77c-d92d3cca88bf admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:05.156670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:05.157269 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:05.157548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:05.157820 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:05.167151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:05.168563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:05.178594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1359/2697] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:05 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:05.599664 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:05.601093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:05.616115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce2a53de-e62d-45ba-a2f8-6fe732287aed None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:05.621491 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:05.622127 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:05.622409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:05.622687 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] PATH_INFO: `/v3/projects/ab48af86528c45a784554040c455be58` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:05.624192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ab48af86528c45a784554040c455be58)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:05.626286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:05.730474 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ab48af86528c45a784554040c455be58', 'request_id': 'req-9d381fbb-b7c8-43f0-9b0b-503420c67094'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:05.735840 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ab48af86528c45a784554040c455be58', 'request_id': 'req-9d381fbb-b7c8-43f0-9b0b-503420c67094'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:05.838903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ab48af86528c45a784554040c455be58', 'request_id': 'req-9d381fbb-b7c8-43f0-9b0b-503420c67094'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:05.851357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] The token cache is being invalidate because project ab48af86528c45a784554040c455be58 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:45:05.851644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:05.852273 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:05.853742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1339/2698] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:45:05 2022] DELETE /identity/v3/projects/ab48af86528c45a784554040c455be58 => generated 0 bytes in 255 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:05.857999 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:05.867880 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87042e66-abbb-45c8-a6e0-1ad276d7de95 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:05.924715 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:05.925354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:05.925630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:05.925898 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] PATH_INFO: `/v3/users/604605e987cd4453a3d9834bf46de8fa` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:05.927038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] RBAC: Authorizing `identity:delete_user(user_id=604605e987cd4453a3d9834bf46de8fa)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:05.928931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:05.930367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:06.224220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '604605e987cd4453a3d9834bf46de8fa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:06.225137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '604605e987cd4453a3d9834bf46de8fa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:06.234307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '604605e987cd4453a3d9834bf46de8fa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:06.235117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '604605e987cd4453a3d9834bf46de8fa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:06.308289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-578246dc-9798-44bf-b42c-17231bd90704 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '604605e987cd4453a3d9834bf46de8fa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:06.343151 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1360/2699] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:45:05 2022] DELETE /identity/v3/users/604605e987cd4453a3d9834bf46de8fa => generated 0 bytes in 486 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:06.788618 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:06.790262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:06.804814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d381fbb-b7c8-43f0-9b0b-503420c67094 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:06.869808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:06.870444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:06.870726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:06.871031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] PATH_INFO: `/v3/projects/581ae10f09bb4800878d721255a3f624` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:06.872139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] RBAC: Authorizing `identity:delete_project(project_id=581ae10f09bb4800878d721255a3f624)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:06.874048 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.183706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '581ae10f09bb4800878d721255a3f624', 'request_id': 'req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:07.189164 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '581ae10f09bb4800878d721255a3f624', 'request_id': 'req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:07.193167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:07.268020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '581ae10f09bb4800878d721255a3f624', 'request_id': 'req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:07.279880 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] The token cache is being invalidate because project 581ae10f09bb4800878d721255a3f624 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:45:07.284029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:07.284029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4ad8c4df-a32b-4fe2-83ff-cec5c6af1f57 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:07.284029 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1340/2700] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:45:06 2022] DELETE /identity/v3/projects/581ae10f09bb4800878d721255a3f624 => generated 0 bytes in 494 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:07.288886 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.289377 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.289544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.308891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:07.310091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:07.369605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1361/2701] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:07.377789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.378076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.378343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.398622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:07.399862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:07.407992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1341/2702] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:07.412392 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:07.422221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bf06f86-023a-46ab-8a10-a2388d212822 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:07.426893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:07.427491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.427766 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.428033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.428669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:45:07.428970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:07.430548 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.446170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1362/2703] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:45:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:45:07.447449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:07.461624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01de172c-5bd5-4f92-a39d-2d08125a9897 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:07.466389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:07.467013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.467290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.467560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.468183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:07.470182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.531787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1342/2704] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:45:07 2022] POST /identity/v3/projects => generated 373 bytes in 85 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:07.536209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:07.546033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea6a17c2-1edf-4309-a86d-b2d9d4ccc9cc admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:07.606144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:07.606751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.607059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.607333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.607989 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:07.609933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.620841 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:07.708476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1363/2705] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:45:07 2022] POST /identity/v3/users => generated 287 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:07.712257 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:07.727508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7911a12-7edb-44a0-b2ca-1e83fa47a0ea admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:07.727508 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:07.727834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.728108 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.728374 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.729006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:07.730584 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.741295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1343/2706] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:07 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:07.751486 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:07.761961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c783d36d-c40d-4dc3-bc43-686208686e17 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:07.767265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93905cfa-ee19-43bf-8433-dc9bc46ab010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:07.768440 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93905cfa-ee19-43bf-8433-dc9bc46ab010 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.768785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93905cfa-ee19-43bf-8433-dc9bc46ab010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.769133 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93905cfa-ee19-43bf-8433-dc9bc46ab010 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.769751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93905cfa-ee19-43bf-8433-dc9bc46ab010 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:07.771397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93905cfa-ee19-43bf-8433-dc9bc46ab010 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.784236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1364/2707] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:07 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:07.789701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:07.799621 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e934854-1d75-4a2f-8930-df21d7de0c81 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:07.804615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:07.805249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.805534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.805809 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] PATH_INFO: `/v3/projects/9ef07a23b7ae4ba49c6c215697d623e5/users/32b0302b32114dbea6292b906c3c3454/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.820339 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=9ef07a23b7ae4ba49c6c215697d623e5, user_id=32b0302b32114dbea6292b906c3c3454, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:07.823018 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:07.941785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1344/2708] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:07 2022] PUT /identity/v3/projects/9ef07a23b7ae4ba49c6c215697d623e5/users/32b0302b32114dbea6292b906c3c3454/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:45:07.946908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba69bed8-4255-4c4d-b843-c8485395b3c2 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:07.947188 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba69bed8-4255-4c4d-b843-c8485395b3c2 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:07.947452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba69bed8-4255-4c4d-b843-c8485395b3c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:07.966455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-ba69bed8-4255-4c4d-b843-c8485395b3c2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:07.967852 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ba69bed8-4255-4c4d-b843-c8485395b3c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:08.026363 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1365/2709] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:08.037077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:08.038967 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:08.097139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95641acd-cde3-49fd-b28a-f831645699fc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:08.100820 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:08.101423 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:08.101702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:08.101973 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:08.111160 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:08.112634 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:08.122355 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1345/2710] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:08 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:09.267292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.273260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.273260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.298108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] MFA Rules not processed for user `32b0302b32114dbea6292b906c3c3454`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:09.299819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:09.380815 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1366/2711] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:09 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:09.385649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:09.396439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-264fe9b9-57ff-4692-a138-65af8a8ef69c service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:09.401596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:09.402315 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.402661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.402971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.403622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:09.405549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:09.502157 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1346/2712] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:45:09 2022] POST /identity/v3/projects => generated 373 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:09.506230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:09.516867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e5dda93-78c4-4713-9403-244577a0f48a None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:09.572948 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:09.573544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.573818 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.574085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.574718 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:09.576642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:09.685186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1367/2713] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:45:09 2022] POST /identity/v3/users => generated 293 bytes in 179 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:09.689794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:09.701078 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebc50bb7-6d3d-48a2-a356-154462090482 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:09.706581 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:09.707218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.707500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.707768 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.708447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:09.710046 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:09.720927 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1347/2714] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:09 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:09.725127 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:09.731544 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:09.737631 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33e06f80-b651-438a-a967-6c2cd23e140b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:09.740342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:09.741014 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.741296 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.741588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] PATH_INFO: `/v3/projects/95264913fc1e44288e1fbe5705682a0a/users/0d460bc8e08f4ceda8e8824b721df549/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.757878 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95264913fc1e44288e1fbe5705682a0a, user_id=0d460bc8e08f4ceda8e8824b721df549, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:09.760456 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:09.851180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1368/2715] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:09 2022] PUT /identity/v3/projects/95264913fc1e44288e1fbe5705682a0a/users/0d460bc8e08f4ceda8e8824b721df549/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:45:09.855208 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:09.866214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20f21cb6-4919-4fe8-bfc9-d5b56de8ceab admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:09.916607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:09.922582 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:09.923222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.923504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.924328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.924965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:09.926552 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:09.936631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1348/2716] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:09 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:09.940969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:09.952858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183c3599-ba25-4e89-9265-36d575eb7cb2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:09.956230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab81f4b8-daa2-4cb1-ac9d-c59e1869ebae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:09.957113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab81f4b8-daa2-4cb1-ac9d-c59e1869ebae admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:09.957239 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab81f4b8-daa2-4cb1-ac9d-c59e1869ebae admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:09.957432 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab81f4b8-daa2-4cb1-ac9d-c59e1869ebae admin admin] PATH_INFO: `/v3/projects/95264913fc1e44288e1fbe5705682a0a/users/0d460bc8e08f4ceda8e8824b721df549/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:09.959399 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab81f4b8-daa2-4cb1-ac9d-c59e1869ebae admin admin] RBAC: Authorizing `identity:create_grant(project_id=95264913fc1e44288e1fbe5705682a0a, user_id=0d460bc8e08f4ceda8e8824b721df549, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:09.962062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab81f4b8-daa2-4cb1-ac9d-c59e1869ebae admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:09.973444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1369/2717] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:09 2022] PUT /identity/v3/projects/95264913fc1e44288e1fbe5705682a0a/users/0d460bc8e08f4ceda8e8824b721df549/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:45:11.250801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:11.252980 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:11.265569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f098cf5-c3a0-4211-9335-f914dbaa4fe7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:11.321393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:11.321873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:11.322260 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:11.325957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:11.326702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:11.332791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:11.332791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] PATH_INFO: `/v3/users/6f40963732654c9fa66cbb85cd7fa0ea` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:11.332791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6f40963732654c9fa66cbb85cd7fa0ea)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:11.332791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:11.365283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] MFA Rules not processed for user `0d460bc8e08f4ceda8e8824b721df549`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:11.366506 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:11.438891 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1370/2718] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:11 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:45:11.450554 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:11.451943 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:11.493205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6f40963732654c9fa66cbb85cd7fa0ea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:11.500306 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6f40963732654c9fa66cbb85cd7fa0ea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:11.500970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6f40963732654c9fa66cbb85cd7fa0ea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:11.501551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6f40963732654c9fa66cbb85cd7fa0ea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:11.509605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d397e352-a7c6-4d3f-846e-8001c06d18a9 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:11.513642 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:11.514247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:11.514528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:11.514957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:11.516710 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:11.547206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5033161e-5e21-451c-b5cf-ca3cfc6adb1e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6f40963732654c9fa66cbb85cd7fa0ea'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:11.569740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1349/2719] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:45:11 2022] DELETE /identity/v3/users/6f40963732654c9fa66cbb85cd7fa0ea => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:11.573889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:11.574225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:11.574528 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:11.578861 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:11.580830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:11.593535 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:11.594109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1371/2720] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:11 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:45:11.594732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:11.658117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1350/2721] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:11 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:45:11.668352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:11.669741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:11.679192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:11.730345 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9721369-75ff-406e-852c-52f45f403310 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:11.781444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:11.782439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:11.782439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:11.782636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:11.792336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:11.793749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:11.803329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1372/2722] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:11 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 135 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:12.234479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:12.238992 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:12.263156 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf381ccd-1fbc-48df-8e41-f987dc70df13 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:12.269459 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:12.270052 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:12.270443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:12.272104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] PATH_INFO: `/v3/projects/914529a2fedc4e699f037c1ef6d0ee40` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:12.272786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=914529a2fedc4e699f037c1ef6d0ee40)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:12.275395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:12.287887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:12.382369 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '914529a2fedc4e699f037c1ef6d0ee40', 'request_id': 'req-def52a76-d142-4772-8b95-0937ea0d90a0'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:12.388076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '914529a2fedc4e699f037c1ef6d0ee40', 'request_id': 'req-def52a76-d142-4772-8b95-0937ea0d90a0'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:12.462055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '914529a2fedc4e699f037c1ef6d0ee40', 'request_id': 'req-def52a76-d142-4772-8b95-0937ea0d90a0'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:12.475226 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] The token cache is being invalidate because project 914529a2fedc4e699f037c1ef6d0ee40 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:45:12.475515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:12.476147 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:12.477812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1351/2723] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:45:12 2022] DELETE /identity/v3/projects/914529a2fedc4e699f037c1ef6d0ee40 => generated 0 bytes in 244 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:13.379786 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:13.381184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:13.437823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5f98657-09e2-4aa9-9113-3df45e8c99fa service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:13.485470 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:13.486062 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:13.486337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:13.486607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:13.488375 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:13.522113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=100 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:13.550145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:13.552321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d446053-00e0-45f0-9022-d95aec7793e7 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:13.562354 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1373/2724] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:13 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 184 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:14.236074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:14.237551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:14.249802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def52a76-d142-4772-8b95-0937ea0d90a0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:14.253593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:14.258301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:14.258301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:14.258301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:14.258301 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:14.315608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:14.317541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:14.326846 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1352/2725] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:14 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:22.993366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:22.993876 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:22.994933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:23.013388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] MFA Rules not processed for user `421aaa238a3c495096dfc55aba76267c`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:23.015483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:23.023491 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1374/2726] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:22 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:24.322362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=49,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:25.404354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:25.405752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:25.496336 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4011cf4-7b1c-4cf8-896a-013af3c48223 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:25.501297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:25.502949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:25.502949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:25.502949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:25.504205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:25.515525 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:25.517087 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:25.527335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1353/2727] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:25 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:32.947894 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:32.959825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4521b862-8cdd-45ae-a264-0a89d81ac785 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:32.964942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:32.965545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:32.965823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:32.966124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:32.967951 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:32.981778 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:32.983547 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:32.995058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1375/2728] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:32 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:33.044765 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:33.046221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:33.111758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7a08f82-466a-4d98-be96-3fcdeba97574 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:33.117055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:33.117670 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:33.117945 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:33.118213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] PATH_INFO: `/v3/users/32b0302b32114dbea6292b906c3c3454` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:33.120206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] RBAC: Authorizing `identity:delete_user(user_id=32b0302b32114dbea6292b906c3c3454)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:33.122140 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:33.321604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '32b0302b32114dbea6292b906c3c3454'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.332270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '32b0302b32114dbea6292b906c3c3454'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.332975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '32b0302b32114dbea6292b906c3c3454'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.334677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '32b0302b32114dbea6292b906c3c3454'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.414350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '32b0302b32114dbea6292b906c3c3454'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.423786 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:33.435926 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1354/2729] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:45:33 2022] DELETE /identity/v3/users/32b0302b32114dbea6292b906c3c3454 => generated 0 bytes in 391 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:33.714892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:33.725893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d62dee02-7c0e-46df-a96c-58787fc266f8 service placement] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:33.783480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:33.784105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:33.784383 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:33.784653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] PATH_INFO: `/v3/projects/9ef07a23b7ae4ba49c6c215697d623e5` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:33.785758 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9ef07a23b7ae4ba49c6c215697d623e5)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:33.787848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:33.906926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9ef07a23b7ae4ba49c6c215697d623e5', 'request_id': 'req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.968086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9ef07a23b7ae4ba49c6c215697d623e5', 'request_id': 'req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.974695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9ef07a23b7ae4ba49c6c215697d623e5', 'request_id': 'req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.985109 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] The token cache is being invalidate because project 9ef07a23b7ae4ba49c6c215697d623e5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:45:33.987746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.987746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:33.987746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1376/2730] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:45:33 2022] DELETE /identity/v3/projects/9ef07a23b7ae4ba49c6c215697d623e5 => generated 0 bytes in 274 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:45:33.992793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:33.994239 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:34.008885 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521a6a4d-076f-4830-a14d-3b8b59dd4d39 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:34.067000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:34.067652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:34.068497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:34.069217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] PATH_INFO: `/v3/users/0d460bc8e08f4ceda8e8824b721df549` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:34.070394 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] RBAC: Authorizing `identity:delete_user(user_id=0d460bc8e08f4ceda8e8824b721df549)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:34.072360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:34.300221 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0d460bc8e08f4ceda8e8824b721df549'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:34.308731 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0d460bc8e08f4ceda8e8824b721df549'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:34.309426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0d460bc8e08f4ceda8e8824b721df549'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:34.310049 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0d460bc8e08f4ceda8e8824b721df549'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:34.392543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-ebd2d3ef-aea1-4bee-97db-83354ae06a2a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0d460bc8e08f4ceda8e8824b721df549'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:34.413340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1355/2731] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:45:33 2022] DELETE /identity/v3/users/0d460bc8e08f4ceda8e8824b721df549 => generated 0 bytes in 421 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:34.760474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:34.762724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:34.776591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07b818ba-1fb1-4646-98d0-d2dc671b6ab6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:34.831315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:34.847842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:34.848442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:34.848727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:34.849048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] PATH_INFO: `/v3/projects/95264913fc1e44288e1fbe5705682a0a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:34.850676 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95264913fc1e44288e1fbe5705682a0a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:34.852639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:34.951415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95264913fc1e44288e1fbe5705682a0a', 'request_id': 'req-c57946ce-3b47-472c-80c1-f5d88b14a7e3'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:35.015173 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95264913fc1e44288e1fbe5705682a0a', 'request_id': 'req-c57946ce-3b47-472c-80c1-f5d88b14a7e3'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:35.022348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95264913fc1e44288e1fbe5705682a0a', 'request_id': 'req-c57946ce-3b47-472c-80c1-f5d88b14a7e3'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:35.033379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] The token cache is being invalidate because project 95264913fc1e44288e1fbe5705682a0a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:45:35.033667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:35.034323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-c57946ce-3b47-472c-80c1-f5d88b14a7e3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:45:35.035935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1377/2732] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:45:34 2022] DELETE /identity/v3/projects/95264913fc1e44288e1fbe5705682a0a => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:35.041642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.041923 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.042246 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.062189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:35.063540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:35.124486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1356/2733] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:35.133251 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.133559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.133831 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.154270 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:35.155507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:35.163474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1378/2734] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:35 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:35.167688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.177531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fa753c6-f28c-4d9b-a2a3-a4e4784550ae None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.182225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.182849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.183125 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.183393 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.184028 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:45:35.184327 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.186053 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.188660 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:35.199011 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1357/2735] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:45:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:45:35.203337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.212942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2809973-ae0b-4aae-a579-f257317b15b4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.217908 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.218500 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.218797 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.219081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.219772 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.221695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.296281 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1379/2736] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:45:35 2022] POST /identity/v3/projects => generated 375 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:35.300444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.310379 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ffc95a2-b9b9-4aca-842a-21d833e2e7e9 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.366349 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.366981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.367256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.367524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.368190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.370449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.444577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1358/2737] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:45:35 2022] POST /identity/v3/users => generated 288 bytes in 145 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:45:35.448885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.459059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81060bca-44d2-4aa4-96c7-99c17ba0760c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.463666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.467481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.467481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.467481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.467481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.467481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.477003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1380/2738] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:35 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:35.480871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.491733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbb88c5f-279b-4386-b021-b327511b0ab7 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.496307 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599017e1-bcb1-411e-a4e4-fcefc7e166c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.496893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599017e1-bcb1-411e-a4e4-fcefc7e166c8 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.497167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599017e1-bcb1-411e-a4e4-fcefc7e166c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.497435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599017e1-bcb1-411e-a4e4-fcefc7e166c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.498064 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599017e1-bcb1-411e-a4e4-fcefc7e166c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.499663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599017e1-bcb1-411e-a4e4-fcefc7e166c8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.511996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1359/2739] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:45:35 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:45:35.516112 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.526191 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9983d506-c38e-4612-aa13-619275c6dfec admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.530744 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.531396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.531671 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.531944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] PATH_INFO: `/v3/projects/f7da72efc3464f59bba9c4c3892dd27e/users/e3caef11f3d64f91a71f29e16d54cc1d/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.545971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7da72efc3464f59bba9c4c3892dd27e, user_id=e3caef11f3d64f91a71f29e16d54cc1d, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.548720 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.649996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1381/2740] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:45:35 2022] PUT /identity/v3/projects/f7da72efc3464f59bba9c4c3892dd27e/users/e3caef11f3d64f91a71f29e16d54cc1d/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:45:35.662148 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.662421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.662689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.691179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] MFA Rules not processed for user `e3caef11f3d64f91a71f29e16d54cc1d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:45:35.692409 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:35.759640 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1360/2741] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:45:35 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:45:35.771705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:35.773453 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:35.837569 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc87ec5b-d47f-42b5-95af-3932a66dd33d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:35.842024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:35.843267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:35.843267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:35.843562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:35.853759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:35.855397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:35.866196 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1382/2742] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:35 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:45.755551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=68,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:45.861450 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=85,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:48.608304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:48.609775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:48.666069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4376f3eb-5194-4a11-b635-f6511931f8b7 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:48.669538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:48.670134 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:48.670412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:48.670685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:48.672451 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:48.734203 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:48.735646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:48.745871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1361/2743] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:48 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 138 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:58.740891 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=50 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:45:59.087520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:59.088975 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:59.100865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed82b509-0e03-42e1-8252-b481d225b28f service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:59.105124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:59.109129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:59.109129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:59.109129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:59.109129 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:59.119422 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:59.121132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:59.129355 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1383/2744] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:59 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:45:59.208933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:45:59.217942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b65929-6a9d-45a6-8865-2e1ec87b927b service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:45:59.221751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:45:59.223789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:45:59.223789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:45:59.223789 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:45:59.224876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:45:59.315798 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:45:59.317256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0327fcd-9f32-4b6a-99e8-b4e7404bdd97 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:45:59.339126 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1362/2745] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:45:59 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:46:00.006549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:46:00.070459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe96326a-a2ee-4126-8090-900a6473a6c8 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:46:00.077776 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:46:00.078950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:46:00.079228 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:46:00.079492 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] PATH_INFO: `/v3/users/e3caef11f3d64f91a71f29e16d54cc1d` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:46:00.082259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e3caef11f3d64f91a71f29e16d54cc1d)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:46:00.086049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:46:00.280978 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e3caef11f3d64f91a71f29e16d54cc1d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:00.281717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e3caef11f3d64f91a71f29e16d54cc1d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:00.288856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e3caef11f3d64f91a71f29e16d54cc1d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:00.289668 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e3caef11f3d64f91a71f29e16d54cc1d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:00.386398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e3caef11f3d64f91a71f29e16d54cc1d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:00.405367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1384/2746] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:46:00 2022] DELETE /identity/v3/users/e3caef11f3d64f91a71f29e16d54cc1d => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:46:00.409807 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:46:00.410260 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:46:00.410385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:46:00.430277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:46:00.433827 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:00.492277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1363/2747] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:46:00 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:46:00.502476 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:46:00.503952 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:00.561198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34f8b810-e69f-4cd4-a0c3-aacd4be8bcb6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:46:00.564984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:46:00.565606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:46:00.565883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:46:00.566152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:46:00.575266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:46:00.576683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:46:00.586032 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1385/2748] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:46:00 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:46:00.976560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:46:00.978175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:00.992242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a9acb31-9e62-4757-a61f-b853c78c9b47 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:46:00.996981 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:46:00.998019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:46:00.998295 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:46:00.998560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] PATH_INFO: `/v3/projects/f7da72efc3464f59bba9c4c3892dd27e` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:46:01.000013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7da72efc3464f59bba9c4c3892dd27e)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:46:01.001931 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:46:01.109107 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7da72efc3464f59bba9c4c3892dd27e', 'request_id': 'req-4523f2cc-b74f-40d6-b326-17e937fc750f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:01.113497 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:46:01.116530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7da72efc3464f59bba9c4c3892dd27e', 'request_id': 'req-4523f2cc-b74f-40d6-b326-17e937fc750f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:01.163635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7da72efc3464f59bba9c4c3892dd27e', 'request_id': 'req-4523f2cc-b74f-40d6-b326-17e937fc750f'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:01.176784 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] The token cache is being invalidate because project f7da72efc3464f59bba9c4c3892dd27e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:46:01.177071 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:01.177734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:46:01.179352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1364/2749] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:46:00 2022] DELETE /identity/v3/projects/f7da72efc3464f59bba9c4c3892dd27e => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:46:10.581828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=85,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:46:11.180829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=2,keystone:SELECT=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:46:32.068705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:46:32.070240 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:32.132114 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecfad8fd-93cf-4700-99b1-d1ef544f1d2b service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:46:32.180856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:46:32.181489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:46:32.181767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:46:32.182054 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:46:32.183830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:32.254374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:46:32.255871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:46:32.266826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1386/2750] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:46:32 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 199 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:46:35.163909 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:46:35.165334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:35.223764 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4523f2cc-b74f-40d6-b326-17e937fc750f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:46:35.227373 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:46:35.227970 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:46:35.228249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:46:35.228529 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:46:35.230225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:46:35.244337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:46:35.245758 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:46:35.257494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1365/2751] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:46:35 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:46:42.261737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=74 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:46:45.252759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=26 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:19.438723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:19.441489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:19.460256 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a9c103d-701b-474b-87f3-749018722304 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:19.466144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:19.469086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:19.469086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:19.469086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:19.470272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:19.483839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:19.485431 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:19.497528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1387/2752] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:19 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 59 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:19.568248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:19.569745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:19.584326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe977b6-f84a-475e-9eb3-35e34b7d700d service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:19.586357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:19.587066 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:19.587344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:19.587616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:19.589419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:19.600294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:19.601753 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed7ab800-3011-4c87-9f22-bf0169546e2f service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:19.610538 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1366/2753] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:19 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 43 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:23.994469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:23.995958 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:24.011627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccebdfe1-b735-41ec-b4e2-d697611a7d80 service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:24.017329 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:24.017937 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:24.018194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:24.018459 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:24.020361 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:24.033297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:24.034897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:24.045591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1388/2754] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:23 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:29.606685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=6 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:30.245308 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:30.245593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:30.245864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:30.267262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] MFA Rules not processed for user `09cb0d1d7888436dae18a45598a8c82a`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:47:30.268501 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:30.341835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1367/2755] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:47:30 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:47:30.663497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:30.673016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65db214d-2052-49ef-a14f-52646b903650 service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:30.676655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:30.679075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:30.679075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:30.679075 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:30.679625 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:30.689323 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:30.690748 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:30.698892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1389/2756] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:30 2022] GET /identity/v3/auth/tokens => generated 3121 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:31.129694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:31.131449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:31.242903 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198923ea-af2b-4295-8278-c878fe08caa5 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:31.250844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:31.251611 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:31.251957 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:31.252288 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:31.254644 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:31.268650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:31.270397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:31.284357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1368/2757] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:31 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 156 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:37.143081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:37.160015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef93ea86-5a70-4469-8fd0-77a570a02d60 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:37.160015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:37.160015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:37.160015 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:37.160502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:37.160502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:37.225763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:37.227248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:37.239858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1390/2758] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:37 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 97 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:41.278344 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=57 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:41.897214 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:41.898941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:41.911685 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-466b8543-7609-492d-8d32-37387e5401d2 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:41.915738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:41.916551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:41.916943 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:41.917293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:41.919184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:41.977142 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:41.996681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf062bc1-0d65-4fa7-85e1-3b050dd88ea0 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:42.002320 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:42.004577 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:42.011535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:42.012414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:42.014205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:42.014723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:42.017734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1369/2759] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 121 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Sep 27 16:47:42.038473 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:42.040061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:42.057101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1391/2760] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:41 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 80 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:42.213172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:42.227626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12d6fdbf-f8af-4f0e-968b-c3abb0f402b6 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:42.233502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:42.234242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:42.234605 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:42.234977 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:42.236780 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:42.252805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:42.255596 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:42.270620 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1370/2761] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:42 2022] GET /identity/v3/auth/tokens => generated 3372 bytes in 62 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:44.787247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:44.792160 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=55 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:44.798214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d952794e-22b4-473e-9e45-235a808e67da service nova] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:44.807723 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:44.808349 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:44.808627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:44.808901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:44.813046 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:44.830592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:44.830592 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:44.840721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1392/2762] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:47:44 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 54 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:47:46.317248 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:46.320424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:46.340770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9550752e-e508-430f-b9d7-875a60719eea service nova] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:46.347562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:46.348222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:46.348507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:46.348783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] PATH_INFO: `/v3/users/421aaa238a3c495096dfc55aba76267c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:46.349982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] RBAC: Authorizing `identity:delete_user(user_id=421aaa238a3c495096dfc55aba76267c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:46.352119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:46.511216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '421aaa238a3c495096dfc55aba76267c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:46.524094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '421aaa238a3c495096dfc55aba76267c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:46.524959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '421aaa238a3c495096dfc55aba76267c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:46.525754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '421aaa238a3c495096dfc55aba76267c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:46.600674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '421aaa238a3c495096dfc55aba76267c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:46.621517 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1371/2763] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:47:46 2022] DELETE /identity/v3/users/421aaa238a3c495096dfc55aba76267c => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:46.944746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:46.956157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5ee2d2-375a-4c9c-bde8-493cf350cefe service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:47.045510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:47.047245 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:47.047767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:47.048305 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] PATH_INFO: `/v3/projects/1e0f894b53284e788cdb4864acd917be` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:47.050510 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1e0f894b53284e788cdb4864acd917be)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:47.053503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:47.225193 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1e0f894b53284e788cdb4864acd917be', 'request_id': 'req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.348135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1e0f894b53284e788cdb4864acd917be', 'request_id': 'req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.362971 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1e0f894b53284e788cdb4864acd917be', 'request_id': 'req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.377508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] The token cache is being invalidate because project 1e0f894b53284e788cdb4864acd917be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:47:47.377508 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.378023 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.379623 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1393/2764] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:47:46 2022] DELETE /identity/v3/projects/1e0f894b53284e788cdb4864acd917be => generated 0 bytes in 435 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:47.383896 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:47.385917 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:47.401469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5641afe-9fce-4ddc-9f4c-18bcf8c6ae31 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:47.474311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:47.483247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:47.485448 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:47.485727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:47.486476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] PATH_INFO: `/v3/users/87b67db042cf48a8ab02e3df17950dbb` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:47.487711 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] RBAC: Authorizing `identity:delete_user(user_id=87b67db042cf48a8ab02e3df17950dbb)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:47.494019 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:47.718447 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87b67db042cf48a8ab02e3df17950dbb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.727480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87b67db042cf48a8ab02e3df17950dbb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.729506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87b67db042cf48a8ab02e3df17950dbb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.730292 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87b67db042cf48a8ab02e3df17950dbb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.787955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87b67db042cf48a8ab02e3df17950dbb'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:47.813229 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1372/2765] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:47:47 2022] DELETE /identity/v3/users/87b67db042cf48a8ab02e3df17950dbb => generated 0 bytes in 430 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:48.076513 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:48.078930 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:48.094002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8e31b7e-57c7-44c3-82e4-a3b3885fdb01 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:48.159083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:48.159994 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:48.160275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:48.160549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] PATH_INFO: `/v3/projects/2dc58006640a43db85dadea3d504c96b` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:48.161814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2dc58006640a43db85dadea3d504c96b)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:48.163801 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:48.188708 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:48.275215 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2dc58006640a43db85dadea3d504c96b', 'request_id': 'req-60939cd9-ec5c-4897-a2d8-04965f6db574'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:48.353969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2dc58006640a43db85dadea3d504c96b', 'request_id': 'req-60939cd9-ec5c-4897-a2d8-04965f6db574'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:48.365276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2dc58006640a43db85dadea3d504c96b', 'request_id': 'req-60939cd9-ec5c-4897-a2d8-04965f6db574'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:48.386303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] The token cache is being invalidate because project 2dc58006640a43db85dadea3d504c96b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:47:48.386721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:48.387552 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:48.392840 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1394/2766] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:47:48 2022] DELETE /identity/v3/projects/2dc58006640a43db85dadea3d504c96b => generated 0 bytes in 315 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:53.131667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:53.133270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:53.149289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e36455a4-5ec6-455b-82d6-e2d458084f1a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:53.212808 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:53.213354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:53.213636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:53.213911 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] PATH_INFO: `/v3/users/af5c944405e8435f9e0eccaaee764cd2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:53.215189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] RBAC: Authorizing `identity:delete_user(user_id=af5c944405e8435f9e0eccaaee764cd2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:53.217182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:53.392122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af5c944405e8435f9e0eccaaee764cd2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:53.399511 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af5c944405e8435f9e0eccaaee764cd2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:53.400196 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af5c944405e8435f9e0eccaaee764cd2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:53.400794 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af5c944405e8435f9e0eccaaee764cd2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:53.463119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af5c944405e8435f9e0eccaaee764cd2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:53.475425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=78,keystone:DELETE=20,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:53.481965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1373/2767] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:47:53 2022] DELETE /identity/v3/users/af5c944405e8435f9e0eccaaee764cd2 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:53.722101 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:53.724116 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:53.739591 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60939cd9-ec5c-4897-a2d8-04965f6db574 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:53.800497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:53.801179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:53.801461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:53.801732 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] PATH_INFO: `/v3/projects/eb77c9fe25cb4abda83cb3d2e665557a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:53.802873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] RBAC: Authorizing `identity:delete_project(project_id=eb77c9fe25cb4abda83cb3d2e665557a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:53.804867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:53.930132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'eb77c9fe25cb4abda83cb3d2e665557a', 'request_id': 'req-5a2033f1-af11-4965-8f7a-e4614933bcfc'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.000337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'eb77c9fe25cb4abda83cb3d2e665557a', 'request_id': 'req-5a2033f1-af11-4965-8f7a-e4614933bcfc'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.007584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'eb77c9fe25cb4abda83cb3d2e665557a', 'request_id': 'req-5a2033f1-af11-4965-8f7a-e4614933bcfc'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.018863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] The token cache is being invalidate because project eb77c9fe25cb4abda83cb3d2e665557a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:47:54.019091 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.019764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.021541 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1395/2768] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:47:53 2022] DELETE /identity/v3/projects/eb77c9fe25cb4abda83cb3d2e665557a => generated 0 bytes in 300 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:54.025748 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:54.027343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:54.040997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1724c9-750a-48d0-b830-08ce63f28a2e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:54.096910 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:54.097533 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:54.097810 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:54.098082 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] PATH_INFO: `/v3/users/88bed0eae4494d248708a57d065287ac` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:54.099219 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88bed0eae4494d248708a57d065287ac)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:54.101130 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:54.305650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88bed0eae4494d248708a57d065287ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.313718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88bed0eae4494d248708a57d065287ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.314434 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88bed0eae4494d248708a57d065287ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.315081 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88bed0eae4494d248708a57d065287ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.394186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88bed0eae4494d248708a57d065287ac'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.412608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1374/2769] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:47:54 2022] DELETE /identity/v3/users/88bed0eae4494d248708a57d065287ac => generated 0 bytes in 387 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:47:54.647568 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:47:54.648976 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:47:54.663474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a2033f1-af11-4965-8f7a-e4614933bcfc admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:47:54.702217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=14,keystone:SELECT=84,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:47:54.720047 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:47:54.720669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:47:54.720942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:47:54.721209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] PATH_INFO: `/v3/projects/7136a363bf5b4311ab7fa5a35013e5aa` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:47:54.722330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=7136a363bf5b4311ab7fa5a35013e5aa)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:47:54.724303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:47:54.834636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '7136a363bf5b4311ab7fa5a35013e5aa', 'request_id': 'req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.931825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '7136a363bf5b4311ab7fa5a35013e5aa', 'request_id': 'req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.938587 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '7136a363bf5b4311ab7fa5a35013e5aa', 'request_id': 'req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.949152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] The token cache is being invalidate because project 7136a363bf5b4311ab7fa5a35013e5aa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:47:54.949443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.950076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:47:54.951589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1396/2770] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:47:54 2022] DELETE /identity/v3/projects/7136a363bf5b4311ab7fa5a35013e5aa => generated 0 bytes in 305 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:48:04.409044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=55,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:48:04.946056 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=33,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:01.433606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:01.435218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:01.448934 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7681d625-1204-45b9-955c-ed337691a3b6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:01.504454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:01.505007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:01.505283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:01.505578 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] PATH_INFO: `/v3/users/09cb0d1d7888436dae18a45598a8c82a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:01.506678 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=09cb0d1d7888436dae18a45598a8c82a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:01.509093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:01.664175 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '09cb0d1d7888436dae18a45598a8c82a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:01.671408 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '09cb0d1d7888436dae18a45598a8c82a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:01.672093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '09cb0d1d7888436dae18a45598a8c82a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:01.672688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '09cb0d1d7888436dae18a45598a8c82a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:01.745350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-9be4f25f-febe-418c-adff-b9016c85dc12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '09cb0d1d7888436dae18a45598a8c82a'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:01.769165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1375/2771] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:01 2022] DELETE /identity/v3/users/09cb0d1d7888436dae18a45598a8c82a => generated 0 bytes in 333 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:02.006259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.011885 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:02.033150 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff8a9d6a-2f96-4cb2-b4b4-f8a9d25fb1b6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.095560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.096860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.097337 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.097612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] PATH_INFO: `/v3/projects/0a89110c26344f608dae59f5616575f3` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.098805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] RBAC: Authorizing `identity:delete_project(project_id=0a89110c26344f608dae59f5616575f3)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.100716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:02.204595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0a89110c26344f608dae59f5616575f3', 'request_id': 'req-0d8d601e-9c7f-41ba-b952-05c44036a33a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:02.304836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0a89110c26344f608dae59f5616575f3', 'request_id': 'req-0d8d601e-9c7f-41ba-b952-05c44036a33a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:02.311941 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0a89110c26344f608dae59f5616575f3', 'request_id': 'req-0d8d601e-9c7f-41ba-b952-05c44036a33a'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:02.322735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] The token cache is being invalidate because project 0a89110c26344f608dae59f5616575f3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:02.323057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:02.323690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-0d8d601e-9c7f-41ba-b952-05c44036a33a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:02.325291 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1397/2772] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:02 2022] DELETE /identity/v3/projects/0a89110c26344f608dae59f5616575f3 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:02.330532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.330837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.331117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.354690 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:02.355937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:02.416559 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1376/2773] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:02.425626 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.425939 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.426210 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.445542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:02.446856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:02.454687 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1398/2774] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:02 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:02.458916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.469191 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4659ec1c-01ff-4960-b274-0967c425536f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.473706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.474296 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.474580 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.474890 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.475532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:02.475838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.478074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:02.489532 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1377/2775] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:49:02.493977 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.502982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a1f86b-2350-4f2a-a2ef-37de1e95fbd8 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.507581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.508155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.508427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.508719 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.509351 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.511263 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:02.625474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1399/2776] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:02 2022] POST /identity/v3/projects => generated 365 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:02.629649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.639706 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14d00810-e287-4471-81bf-e26e1a92759d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.643294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:02.698068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.698663 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.698974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.699249 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.699878 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.701853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:02.836192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1378/2777] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:02 2022] POST /identity/v3/users => generated 283 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:49:02.840814 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.851077 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7151c9a-a5c8-40d7-b7c4-dc91044ac0c5 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.855628 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.856216 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.856494 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.856762 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.857396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.859018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:02.868980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1400/2778] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:02 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:02.872913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.889133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a77ba916-6a83-4a5f-9b79-c84ad2f569c8 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.889133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06c993ae-347b-44be-adc7-2154a180ed7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.889133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c993ae-347b-44be-adc7-2154a180ed7a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.889133 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c993ae-347b-44be-adc7-2154a180ed7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.889597 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06c993ae-347b-44be-adc7-2154a180ed7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.889871 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c993ae-347b-44be-adc7-2154a180ed7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.891531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06c993ae-347b-44be-adc7-2154a180ed7a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:02.902668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1379/2779] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:02 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:02.906649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:02.916745 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b482e23-fa43-46b5-8d72-b60aabe814c2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:02.921293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:02.921926 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:02.922209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:02.922481 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] PATH_INFO: `/v3/projects/615f56e525b24f37b9056461f4b2ebcc/users/b8a9fefea1d94be5ba589a2a2294ce5d/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:02.936957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] RBAC: Authorizing `identity:create_grant(project_id=615f56e525b24f37b9056461f4b2ebcc, user_id=b8a9fefea1d94be5ba589a2a2294ce5d, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:02.939655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:03.028143 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1401/2780] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:02 2022] PUT /identity/v3/projects/615f56e525b24f37b9056461f4b2ebcc/users/b8a9fefea1d94be5ba589a2a2294ce5d/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:03.038328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:03.038629 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:03.038953 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:03.067150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] MFA Rules not processed for user `b8a9fefea1d94be5ba589a2a2294ce5d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:03.068357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:03.132184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1380/2781] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:03 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:03.144044 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:03.145677 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:03.175783 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:03.203039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b171c7-8b60-4ad8-aa28-f65d7985ac94 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:03.206857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a7cb9af-016e-4819-87c0-736af516f8f2 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:03.207452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cb9af-016e-4819-87c0-736af516f8f2 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:03.207727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cb9af-016e-4819-87c0-736af516f8f2 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:03.207993 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a7cb9af-016e-4819-87c0-736af516f8f2 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:03.216471 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7cb9af-016e-4819-87c0-736af516f8f2 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:03.217888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a7cb9af-016e-4819-87c0-736af516f8f2 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:03.226764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1402/2782] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:03 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:13.097733 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:13.158654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b00a4fc-654a-4ff6-8023-246d2c9d8c99 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:13.163114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:13.163730 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:13.164006 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:13.164274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] PATH_INFO: `/v3/users/b8a9fefea1d94be5ba589a2a2294ce5d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:13.165365 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] RBAC: Authorizing `identity:delete_user(user_id=b8a9fefea1d94be5ba589a2a2294ce5d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:13.167303 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:13.182433 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=3,keystone:DELETE=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:13.275787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=14 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:13.344653 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b8a9fefea1d94be5ba589a2a2294ce5d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:13.351749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b8a9fefea1d94be5ba589a2a2294ce5d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:13.352420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b8a9fefea1d94be5ba589a2a2294ce5d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:13.353096 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b8a9fefea1d94be5ba589a2a2294ce5d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:13.415702 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b8a9fefea1d94be5ba589a2a2294ce5d'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:13.433884 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1381/2783] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:13 2022] DELETE /identity/v3/users/b8a9fefea1d94be5ba589a2a2294ce5d => generated 0 bytes in 337 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:13.439037 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:13.439325 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:13.439608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:13.458985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:13.460176 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:13.516680 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1403/2784] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:13 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:13.526833 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:13.528225 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:13.585263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eebc1589-8e9a-4ddd-bb53-46392eee0f6d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:13.589117 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d260bbb1-8342-4ba5-850b-ba582fcadbb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:13.589727 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260bbb1-8342-4ba5-850b-ba582fcadbb7 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:13.590007 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260bbb1-8342-4ba5-850b-ba582fcadbb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:13.590311 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d260bbb1-8342-4ba5-850b-ba582fcadbb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:13.599626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d260bbb1-8342-4ba5-850b-ba582fcadbb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:13.601058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d260bbb1-8342-4ba5-850b-ba582fcadbb7 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:13.610829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1382/2785] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:13 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:13.991824 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:13.993249 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:14.005933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0c375ee-f247-4869-a397-acc51ad73ddd None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.010563 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.011229 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.011503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.011767 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] PATH_INFO: `/v3/projects/615f56e525b24f37b9056461f4b2ebcc` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.012860 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] RBAC: Authorizing `identity:delete_project(project_id=615f56e525b24f37b9056461f4b2ebcc)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.015017 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.118711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '615f56e525b24f37b9056461f4b2ebcc', 'request_id': 'req-9568ecbd-33e2-4ea4-b28f-3ce05644b326'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:14.200845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '615f56e525b24f37b9056461f4b2ebcc', 'request_id': 'req-9568ecbd-33e2-4ea4-b28f-3ce05644b326'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:14.207659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '615f56e525b24f37b9056461f4b2ebcc', 'request_id': 'req-9568ecbd-33e2-4ea4-b28f-3ce05644b326'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:14.218439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] The token cache is being invalidate because project 615f56e525b24f37b9056461f4b2ebcc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:14.218717 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:14.219371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9568ecbd-33e2-4ea4-b28f-3ce05644b326 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:14.220842 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1404/2786] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:13 2022] DELETE /identity/v3/projects/615f56e525b24f37b9056461f4b2ebcc => generated 0 bytes in 230 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:49:14.226195 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.226671 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.226825 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.246971 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:14.248317 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:14.306314 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1383/2787] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:14.313929 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.314416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.314522 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.334330 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:14.335588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:14.344266 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1405/2788] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:14 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:14.348432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.358734 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ca81c0a-8bd3-4150-bf99-2c2709a68544 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.363326 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.363916 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.364192 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.364462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.365098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:14.365400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.367020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.378419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1384/2789] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:14 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:49:14.382537 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.392016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ef6a8e3-836b-4e24-a29e-3cf5e60e7fc0 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.397200 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.397790 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.398069 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.398342 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.399041 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.400938 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.508009 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1406/2790] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:14 2022] POST /identity/v3/projects => generated 373 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:14.512413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.523098 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c17d6dde-02ea-430f-ad62-7ab3b40a20c4 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.543835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=9,keystone:SELECT=90,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:14.581259 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.582216 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.582540 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.582836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.583452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.585370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.684385 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1385/2791] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:14 2022] POST /identity/v3/users => generated 287 bytes in 169 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:14.685683 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.696199 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd6b7db4-6046-4675-934a-bf647ee99433 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.701602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.702187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.702534 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.702827 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.703460 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.705039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.714802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1407/2792] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:14.719350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.729386 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0801fdc-9c5f-4796-a4ce-ef863c795773 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.734526 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a95c5a-5c67-4da9-b621-7b46f68558b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.735152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a95c5a-5c67-4da9-b621-7b46f68558b7 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.735430 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a95c5a-5c67-4da9-b621-7b46f68558b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.735700 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a95c5a-5c67-4da9-b621-7b46f68558b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.736334 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a95c5a-5c67-4da9-b621-7b46f68558b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.737948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a95c5a-5c67-4da9-b621-7b46f68558b7 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.748324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1386/2793] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:14 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:14.752403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.762147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4138f56b-2e2e-4bb0-bf09-4cc4351c45b8 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:14.767899 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:14.768799 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.769076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.769350 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] PATH_INFO: `/v3/projects/55b3f15fb2714f2cb893a0f4430d9bac/users/0eb34e423bd84c8e808a61e8483a68b7/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.784632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] RBAC: Authorizing `identity:create_grant(project_id=55b3f15fb2714f2cb893a0f4430d9bac, user_id=0eb34e423bd84c8e808a61e8483a68b7, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:14.787318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:14.852590 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1408/2794] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:14 2022] PUT /identity/v3/projects/55b3f15fb2714f2cb893a0f4430d9bac/users/0eb34e423bd84c8e808a61e8483a68b7/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:14.867975 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:14.868258 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:14.868606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:14.897352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] MFA Rules not processed for user `0eb34e423bd84c8e808a61e8483a68b7`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:14.898589 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:14.963093 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1387/2795] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:14.974645 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:14.978873 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:14.999638 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:15.052049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e0f5a2f-8c56-464e-a5bd-beabfa008d43 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:15.057624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89fa2c15-d764-4d41-b823-4bcf288ad61f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'ab556e3087544c42929aab1b35d145a4', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:15.058230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fa2c15-d764-4d41-b823-4bcf288ad61f service nova] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:15.058511 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fa2c15-d764-4d41-b823-4bcf288ad61f service nova] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:15.058803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89fa2c15-d764-4d41-b823-4bcf288ad61f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:15.067427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fa2c15-d764-4d41-b823-4bcf288ad61f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:15.069088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89fa2c15-d764-4d41-b823-4bcf288ad61f service nova] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:15.079769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1409/2796] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:15.286555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:15.347958 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d81dc64-7c0e-4925-918e-8745cb280b73 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:15.352406 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:15.353026 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:15.353300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:15.353569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] PATH_INFO: `/v3/users/0eb34e423bd84c8e808a61e8483a68b7` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:15.354673 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0eb34e423bd84c8e808a61e8483a68b7)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:15.356613 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:15.548732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:INSERT=3,keystone:DELETE=5 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:15.558185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0eb34e423bd84c8e808a61e8483a68b7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:15.565506 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0eb34e423bd84c8e808a61e8483a68b7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:15.566178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0eb34e423bd84c8e808a61e8483a68b7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:15.566800 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0eb34e423bd84c8e808a61e8483a68b7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:15.629212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0eb34e423bd84c8e808a61e8483a68b7'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:15.647962 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1388/2797] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:15 2022] DELETE /identity/v3/users/0eb34e423bd84c8e808a61e8483a68b7 => generated 0 bytes in 362 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:15.652648 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:15.653179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:15.653286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:15.672773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:15.674034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:15.731887 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1410/2798] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:15 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:15.741818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:15.743233 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:15.799626 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f807e68-2fcb-4082-81f8-37c1ca5ba079 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:15.803165 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03c4df7b-9a98-4487-bb13-fe9408ac0964 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:15.803755 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c4df7b-9a98-4487-bb13-fe9408ac0964 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:15.804030 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c4df7b-9a98-4487-bb13-fe9408ac0964 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:15.804300 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03c4df7b-9a98-4487-bb13-fe9408ac0964 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:15.813887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c4df7b-9a98-4487-bb13-fe9408ac0964 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:15.815363 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03c4df7b-9a98-4487-bb13-fe9408ac0964 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:15.824883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1389/2799] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:15 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:16.326328 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:16.327738 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:16.343420 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-399cad47-0cc2-4ec3-b1d7-791ae67b7b7b None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:16.348409 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:16.349192 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:16.349475 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:16.349742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] PATH_INFO: `/v3/projects/55b3f15fb2714f2cb893a0f4430d9bac` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:16.350932 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] RBAC: Authorizing `identity:delete_project(project_id=55b3f15fb2714f2cb893a0f4430d9bac)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:16.352984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:16.473018 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '55b3f15fb2714f2cb893a0f4430d9bac', 'request_id': 'req-413006b8-ad95-4462-84b5-ba22dce377aa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:16.555443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '55b3f15fb2714f2cb893a0f4430d9bac', 'request_id': 'req-413006b8-ad95-4462-84b5-ba22dce377aa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:16.562264 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '55b3f15fb2714f2cb893a0f4430d9bac', 'request_id': 'req-413006b8-ad95-4462-84b5-ba22dce377aa'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:16.572627 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] The token cache is being invalidate because project 55b3f15fb2714f2cb893a0f4430d9bac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:16.572915 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:16.573550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-413006b8-ad95-4462-84b5-ba22dce377aa admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:16.575070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1411/2800] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:16 2022] DELETE /identity/v3/projects/55b3f15fb2714f2cb893a0f4430d9bac => generated 0 bytes in 250 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:16.580372 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:16.580850 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:16.580954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:16.600864 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:16.602068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:16.660895 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1390/2801] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:16.668595 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:16.669078 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:16.669181 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:16.689403 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:16.690594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:16.698416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1412/2802] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:16 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:16.702524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:16.712068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd40cde8-82a0-4925-8c37-23be7a90427f None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:16.717534 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:16.718119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:16.718399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:16.718672 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] PATH_INFO: `/v3/domains` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:16.719351 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81613) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:16.719654 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:16.721234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:16.733399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1391/2803] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:49:16.737615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:16.746658 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-705a901e-d8c1-438e-a160-910c81d8c535 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:16.751722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:16.752304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:16.752580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:16.752848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:16.753469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:16.755396 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:16.857074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:16.863326 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1413/2804] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:16 2022] POST /identity/v3/projects => generated 403 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:16.867661 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:16.878212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48356005-f6e4-4ee8-ab8b-6ad6c6185425 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:16.920015 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=94,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:16.936242 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:16.936876 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:16.937179 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:16.937453 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:16.938086 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:16.940069 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:17.050414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1392/2805] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:16 2022] POST /identity/v3/users => generated 302 bytes in 183 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:17.054593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:17.065636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a234cad4-ccf5-436d-8b5d-88a7509dc8bd admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:17.070144 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:17.070734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:17.071039 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:17.071307 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:17.071933 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:17.073515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:17.083897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1414/2806] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:17 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:17.087915 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:17.099073 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-405b32f5-8be8-485b-90ea-55f31e307d29 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:17.103796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b74d5950-9b75-4d83-8f52-8066459b2585 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:17.104397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d5950-9b75-4d83-8f52-8066459b2585 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:17.104683 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d5950-9b75-4d83-8f52-8066459b2585 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:17.104942 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b74d5950-9b75-4d83-8f52-8066459b2585 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:17.105593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74d5950-9b75-4d83-8f52-8066459b2585 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:17.107749 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b74d5950-9b75-4d83-8f52-8066459b2585 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:17.118262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1393/2807] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:17 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:17.122302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:17.132283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9245a46-552c-498a-a1a9-1cda0c5d71e2 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:17.136796 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:17.137428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:17.137706 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:17.137981 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] PATH_INFO: `/v3/projects/5121ac4bf7dc4247b7dc637049dcaa20/users/50dcfcfb197b46e1be09866acc0eb0a2/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:17.153629 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5121ac4bf7dc4247b7dc637049dcaa20, user_id=50dcfcfb197b46e1be09866acc0eb0a2, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:17.156897 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:17.301163 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1415/2808] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:17 2022] PUT /identity/v3/projects/5121ac4bf7dc4247b7dc637049dcaa20/users/50dcfcfb197b46e1be09866acc0eb0a2/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:17.306189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f8afb-a321-4c06-be9a-e2f6e3e2a725 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:17.306472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f8afb-a321-4c06-be9a-e2f6e3e2a725 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:17.306744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b2f8afb-a321-4c06-be9a-e2f6e3e2a725 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:17.327067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-6b2f8afb-a321-4c06-be9a-e2f6e3e2a725 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:17.328274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-6b2f8afb-a321-4c06-be9a-e2f6e3e2a725 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:17.386912 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1394/2809] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:17 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:49:17.397580 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:17.399367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:17.457369 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02b811fe-549c-4080-9576-9a4eb4c9da23 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:17.460965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:17.461562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:17.461832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:17.462095 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:17.471787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:17.473230 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:17.483411 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1416/2810] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:17 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:18.752543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:18.753231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:18.753513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:18.785624 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] MFA Rules not processed for user `50dcfcfb197b46e1be09866acc0eb0a2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:18.787531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:18.853387 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1395/2811] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:18 2022] POST /identity/v3/auth/tokens => generated 3280 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:18.857980 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:18.867806 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c34e685-8c4e-4837-8361-b17656142feb service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:18.873000 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:18.873641 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:18.873920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:18.874182 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] PATH_INFO: `/v3/projects` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:18.874847 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:18.876756 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:19.006003 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1417/2812] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:18 2022] POST /identity/v3/projects => generated 403 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:49:19.009989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:19.021462 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe8171-7434-44ab-aa6d-320a1add02b6 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:19.067623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:19.077360 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:19.077959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:19.078234 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:19.078504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] PATH_INFO: `/v3/users` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:19.079170 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:19.081114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:19.170370 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1396/2813] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:19 2022] POST /identity/v3/users => generated 308 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:19.174364 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:19.184833 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dca5600-9330-4ffe-9181-646f115af505 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:19.190226 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:19.190839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:19.191117 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:19.191385 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:19.192043 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:19.193659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:19.205358 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1418/2814] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:19 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:19.209479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:19.219631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d422875d-98fb-4213-bc78-d0c479e5f0e1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:19.224466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:19.225122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:19.225429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:19.225841 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] PATH_INFO: `/v3/projects/c46adf70d1314334931533a1201a10af/users/f0d7493e0d9e4951bcb4e4ecd8d85514/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:19.240849 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c46adf70d1314334931533a1201a10af, user_id=f0d7493e0d9e4951bcb4e4ecd8d85514, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:19.243575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:19.323901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1397/2815] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:19 2022] PUT /identity/v3/projects/c46adf70d1314334931533a1201a10af/users/f0d7493e0d9e4951bcb4e4ecd8d85514/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:19.327819 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:19.337917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b44bf90-2a89-4e57-9a22-0100363a8927 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:19.393919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82e6c61d-fa15-459a-9af0-6681a84993e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:19.394507 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e6c61d-fa15-459a-9af0-6681a84993e7 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:19.394798 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e6c61d-fa15-459a-9af0-6681a84993e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:19.395070 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82e6c61d-fa15-459a-9af0-6681a84993e7 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:19.395734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e6c61d-fa15-459a-9af0-6681a84993e7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:19.397289 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82e6c61d-fa15-459a-9af0-6681a84993e7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:19.407258 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1419/2816] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:19 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:19.411573 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:19.422504 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7ea95a7-51e5-4410-a327-ebb6818748b0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:19.427218 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:19.427862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:19.428139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:19.428439 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] PATH_INFO: `/v3/projects/c46adf70d1314334931533a1201a10af/users/f0d7493e0d9e4951bcb4e4ecd8d85514/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:19.430341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c46adf70d1314334931533a1201a10af, user_id=f0d7493e0d9e4951bcb4e4ecd8d85514, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:19.432995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:19.443155 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1398/2817] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:19 2022] PUT /identity/v3/projects/c46adf70d1314334931533a1201a10af/users/f0d7493e0d9e4951bcb4e4ecd8d85514/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:20.610458 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:20.611002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:20.611107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:20.642741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] MFA Rules not processed for user `f0d7493e0d9e4951bcb4e4ecd8d85514`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:20.644272 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:20.662739 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:20.716501 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1420/2818] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:20 2022] POST /identity/v3/auth/tokens => generated 3347 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:21.928561 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:21.930329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:22.004874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd63c472-6547-4831-891c-6598f01db5c6 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:22.010608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:22.011250 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:22.011524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:22.011793 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:22.014094 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:22.026604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:22.028200 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:22.038732 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1399/2819] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:21 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 111 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:22.090843 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:22.158064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6d6961c-0b15-48e9-a730-6f80db04e824 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:22.162643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:22.163300 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:22.163576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:22.163845 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] PATH_INFO: `/v3/users/50dcfcfb197b46e1be09866acc0eb0a2` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:22.164957 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] RBAC: Authorizing `identity:delete_user(user_id=50dcfcfb197b46e1be09866acc0eb0a2)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:22.166896 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:22.361589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '50dcfcfb197b46e1be09866acc0eb0a2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.362184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '50dcfcfb197b46e1be09866acc0eb0a2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.369318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '50dcfcfb197b46e1be09866acc0eb0a2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.369979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '50dcfcfb197b46e1be09866acc0eb0a2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.429074 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '50dcfcfb197b46e1be09866acc0eb0a2'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.448038 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1421/2820] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:22 2022] DELETE /identity/v3/users/50dcfcfb197b46e1be09866acc0eb0a2 => generated 0 bytes in 358 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:22.710999 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:22.722016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fddc3fb0-12a9-49c2-8a6a-88845c3de3b6 service placement] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:22.780858 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:22.781523 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:22.781802 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:22.782077 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] PATH_INFO: `/v3/projects/5121ac4bf7dc4247b7dc637049dcaa20` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:22.783247 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5121ac4bf7dc4247b7dc637049dcaa20)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:22.785206 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:22.872032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=93,keystone:INSERT=5,keystone:DELETE=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:22.882837 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5121ac4bf7dc4247b7dc637049dcaa20', 'request_id': 'req-471e2abf-5d01-4151-802e-fbd76d24a197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.888268 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5121ac4bf7dc4247b7dc637049dcaa20', 'request_id': 'req-471e2abf-5d01-4151-802e-fbd76d24a197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.966236 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5121ac4bf7dc4247b7dc637049dcaa20', 'request_id': 'req-471e2abf-5d01-4151-802e-fbd76d24a197'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.977608 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] The token cache is being invalidate because project 5121ac4bf7dc4247b7dc637049dcaa20 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:22.981173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.981173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:22.981173 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1400/2821] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:22 2022] DELETE /identity/v3/projects/5121ac4bf7dc4247b7dc637049dcaa20 => generated 0 bytes in 270 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:22.985610 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:22.987092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:23.001120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01f19c27-face-48fb-81b5-8a27219f71cf admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:23.032764 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:23.061802 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:23.062455 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:23.062734 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:23.063033 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] PATH_INFO: `/v3/users/f0d7493e0d9e4951bcb4e4ecd8d85514` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:23.064152 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f0d7493e0d9e4951bcb4e4ecd8d85514)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:23.066061 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:23.271944 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f0d7493e0d9e4951bcb4e4ecd8d85514'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.272576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f0d7493e0d9e4951bcb4e4ecd8d85514'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.280020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f0d7493e0d9e4951bcb4e4ecd8d85514'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.280637 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f0d7493e0d9e4951bcb4e4ecd8d85514'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.355211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-55e26bee-9422-49eb-93d2-dd2ecf8e1ca6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f0d7493e0d9e4951bcb4e4ecd8d85514'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.374170 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1422/2822] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:22 2022] DELETE /identity/v3/users/f0d7493e0d9e4951bcb4e4ecd8d85514 => generated 0 bytes in 389 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:23.614667 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:23.616309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:23.631189 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471e2abf-5d01-4151-802e-fbd76d24a197 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:23.689914 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:23.690542 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:23.690848 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:23.691210 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] PATH_INFO: `/v3/projects/c46adf70d1314334931533a1201a10af` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:23.692436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c46adf70d1314334931533a1201a10af)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:23.696545 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:23.836103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c46adf70d1314334931533a1201a10af', 'request_id': 'req-8549063a-c3a5-4da2-ae93-7df65723ea24'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.842252 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c46adf70d1314334931533a1201a10af', 'request_id': 'req-8549063a-c3a5-4da2-ae93-7df65723ea24'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.929698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c46adf70d1314334931533a1201a10af', 'request_id': 'req-8549063a-c3a5-4da2-ae93-7df65723ea24'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.942092 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] The token cache is being invalidate because project c46adf70d1314334931533a1201a10af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:23.942401 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.943044 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-8549063a-c3a5-4da2-ae93-7df65723ea24 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:23.944505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1401/2823] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:23 2022] DELETE /identity/v3/projects/c46adf70d1314334931533a1201a10af => generated 0 bytes in 331 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:23.949528 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:23.949811 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:23.950087 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:23.970550 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:23.971784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:24.031050 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1423/2824] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:23 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:24.038615 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.039115 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.039217 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.058699 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:24.060516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:24.068335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1402/2825] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:24 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:24.072483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.082807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bfa8529-e692-4984-8854-7a082d01a0e4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.087380 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.087973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.088250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.088518 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.089158 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:24.089489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.091211 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.102695 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1424/2826] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:24 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:49:24.107023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.116240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e83523-abc3-4a20-b351-b9680b5be9f4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.121061 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.121649 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.122023 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.122838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.123473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.125396 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.218829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1403/2827] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:24 2022] POST /identity/v3/projects => generated 363 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:24.222854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.232914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94a856a5-3882-4778-9b80-8ddb73352251 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.273088 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:24.288893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.289480 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.289759 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.290027 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.290655 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.292614 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.383913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1425/2828] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:24 2022] POST /identity/v3/users => generated 282 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:24.387883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.398656 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffac56dc-bdc4-498f-b14d-0f2cf74c1010 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.403381 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.404067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.404329 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.404604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.405270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.406902 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.416834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1404/2829] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:24 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:24.421255 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.431136 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bb122e-dfca-4e54-94b9-dc99f6d072d7 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.436578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97edf215-e067-4041-b17f-73e03c8a74f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.437157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edf215-e067-4041-b17f-73e03c8a74f6 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.437433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edf215-e067-4041-b17f-73e03c8a74f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.437701 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97edf215-e067-4041-b17f-73e03c8a74f6 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.438327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97edf215-e067-4041-b17f-73e03c8a74f6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.440059 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97edf215-e067-4041-b17f-73e03c8a74f6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.450429 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1426/2830] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:24 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:24.454516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.464625 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b936986-c5c5-4ac9-8c67-08d120151adc admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.469357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.469937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.470213 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.470481 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] PATH_INFO: `/v3/projects/33f4a96fbb054389be440960061d7292/users/8cd6e9180cd14625b86bc4bdaac3b282/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.482770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:24.486688 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=33f4a96fbb054389be440960061d7292, user_id=8cd6e9180cd14625b86bc4bdaac3b282, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.489502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.583814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1405/2831] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:24 2022] PUT /identity/v3/projects/33f4a96fbb054389be440960061d7292/users/8cd6e9180cd14625b86bc4bdaac3b282/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 130 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:24.650108 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.650384 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.650649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.680542 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] MFA Rules not processed for user `8cd6e9180cd14625b86bc4bdaac3b282`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:24.681757 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:24.746659 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1427/2832] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:24 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:24.751103 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.812701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88ccc725-ea52-47a1-8f69-f70e84bc1b5a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.817458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.818059 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.818341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.818616 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.819284 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.821201 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:24.911088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1406/2833] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:24 2022] POST /identity/v3/projects => generated 363 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:24.915304 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:24.926428 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5109291-bb6b-4530-a994-fd4effea7902 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:24.980503 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:24.981107 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:24.981389 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:24.982214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:24.982883 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:24.984800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.060917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1428/2834] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:24 2022] POST /identity/v3/users => generated 288 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:25.064812 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:25.075843 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2180eeb5-cc02-4c57-ac80-a2bcb3e8544d admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:25.080692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:25.081289 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.081565 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.081834 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.082500 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:25.084163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.095738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1407/2835] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:25 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:25.099890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:25.109970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d10fe033-991e-4b44-88ab-318a0555670c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:25.114643 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:25.115315 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.115593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.115865 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] PATH_INFO: `/v3/projects/506b78dcbadf409da9825aa26cd076bd/users/28ab5ee2eeaf4e2a8644b087bdd21707/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.130334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] RBAC: Authorizing `identity:create_grant(project_id=506b78dcbadf409da9825aa26cd076bd, user_id=28ab5ee2eeaf4e2a8644b087bdd21707, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:25.133002 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.209735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1429/2836] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:25 2022] PUT /identity/v3/projects/506b78dcbadf409da9825aa26cd076bd/users/28ab5ee2eeaf4e2a8644b087bdd21707/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 110 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:25.213759 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:25.224472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1083b4c-3aca-46f5-b94d-3884e105a839 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:25.280674 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0025af2a-0f04-4a88-ac8f-412efdd8f97f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:25.281275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0025af2a-0f04-4a88-ac8f-412efdd8f97f admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.281551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0025af2a-0f04-4a88-ac8f-412efdd8f97f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.281816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0025af2a-0f04-4a88-ac8f-412efdd8f97f admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.282440 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0025af2a-0f04-4a88-ac8f-412efdd8f97f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:25.284068 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0025af2a-0f04-4a88-ac8f-412efdd8f97f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.293560 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1408/2837] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:25 2022] GET /identity/v3/roles => generated 1420 bytes in 80 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:25.297919 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:25.308113 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e8b335e-02ec-40d8-950a-71e65877015b admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:25.312864 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:25.313449 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.313724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.313996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] PATH_INFO: `/v3/projects/506b78dcbadf409da9825aa26cd076bd/users/28ab5ee2eeaf4e2a8644b087bdd21707/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.315914 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] RBAC: Authorizing `identity:create_grant(project_id=506b78dcbadf409da9825aa26cd076bd, user_id=28ab5ee2eeaf4e2a8644b087bdd21707, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:25.318564 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.329779 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1430/2838] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:25 2022] PUT /identity/v3/projects/506b78dcbadf409da9825aa26cd076bd/users/28ab5ee2eeaf4e2a8644b087bdd21707/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:25.339587 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.339904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.344040 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.367836 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] MFA Rules not processed for user `28ab5ee2eeaf4e2a8644b087bdd21707`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:25.369033 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:25.416515 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:25.438522 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1409/2839] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:25 2022] POST /identity/v3/auth/tokens => generated 3307 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:25.449490 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:25.451741 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:25.457135 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:25.511068 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766f0059-5df7-4df7-a7c6-4735b58ed482 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:25.515588 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:25.516278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.516549 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.516812 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.525866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:25.527394 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.536773 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1431/2840] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:25 2022] GET /identity/v3/auth/tokens => generated 3307 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:25.734494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:25.744205 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-277dba2e-aacb-4cbd-8c5d-740ea39aec52 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:25.748796 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-432cbc05-8bc9-436f-b1a9-76bf0ea71180 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:25.749397 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432cbc05-8bc9-436f-b1a9-76bf0ea71180 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:25.749698 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432cbc05-8bc9-436f-b1a9-76bf0ea71180 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:25.749955 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432cbc05-8bc9-436f-b1a9-76bf0ea71180 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:25.814857 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432cbc05-8bc9-436f-b1a9-76bf0ea71180 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:25.816297 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-432cbc05-8bc9-436f-b1a9-76bf0ea71180 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:25.825472 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1410/2841] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:25 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:29.452735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:29.513347 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f12bfdb-5f4e-455c-836a-020566e6dd77 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:29.518209 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:29.519231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:29.519231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:29.519499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] PATH_INFO: `/v3/users/8cd6e9180cd14625b86bc4bdaac3b282` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:29.520468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8cd6e9180cd14625b86bc4bdaac3b282)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:29.522360 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:29.720912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8cd6e9180cd14625b86bc4bdaac3b282'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:29.721568 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8cd6e9180cd14625b86bc4bdaac3b282'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:29.728555 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8cd6e9180cd14625b86bc4bdaac3b282'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:29.729198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8cd6e9180cd14625b86bc4bdaac3b282'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:29.803515 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8cd6e9180cd14625b86bc4bdaac3b282'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:29.821636 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1432/2842] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:29 2022] DELETE /identity/v3/users/8cd6e9180cd14625b86bc4bdaac3b282 => generated 0 bytes in 369 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:29.826264 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:29.829076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:29.829076 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:29.846941 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:29.848141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:29.907032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1411/2843] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:29 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:29.916724 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:29.918248 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:29.946339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:29.975303 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d120c22-ea9a-41c3-9e0e-71fa21c421d4 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:29.979106 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:29.979705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:29.979982 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:29.980735 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:29.990448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:29.991917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:30.001931 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1433/2844] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:29 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:30.408402 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:30.410555 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:30.423337 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbe6f1a8-ea1c-484d-a021-0d7743498afb None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:30.427994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:30.428627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:30.428901 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:30.429171 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] PATH_INFO: `/v3/projects/33f4a96fbb054389be440960061d7292` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:30.430286 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33f4a96fbb054389be440960061d7292)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:30.432276 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:30.532816 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33f4a96fbb054389be440960061d7292', 'request_id': 'req-03fdfde6-b807-45b5-803e-ff1520267b31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.538485 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33f4a96fbb054389be440960061d7292', 'request_id': 'req-03fdfde6-b807-45b5-803e-ff1520267b31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.611612 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33f4a96fbb054389be440960061d7292', 'request_id': 'req-03fdfde6-b807-45b5-803e-ff1520267b31'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.623703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] The token cache is being invalidate because project 33f4a96fbb054389be440960061d7292 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:30.623703 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.624716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.626479 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1412/2845] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:30 2022] DELETE /identity/v3/projects/33f4a96fbb054389be440960061d7292 => generated 0 bytes in 219 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:30.630858 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:30.640410 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09ad33d3-53fc-4770-963d-1ad02251a1ec service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:30.697438 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:30.698093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:30.698366 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:30.698632 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] PATH_INFO: `/v3/users/28ab5ee2eeaf4e2a8644b087bdd21707` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:30.699789 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] RBAC: Authorizing `identity:delete_user(user_id=28ab5ee2eeaf4e2a8644b087bdd21707)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:30.701727 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:30.851906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '28ab5ee2eeaf4e2a8644b087bdd21707'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.852561 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '28ab5ee2eeaf4e2a8644b087bdd21707'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.859583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '28ab5ee2eeaf4e2a8644b087bdd21707'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.860203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '28ab5ee2eeaf4e2a8644b087bdd21707'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.914743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9e40f6a7-cc3b-42d6-a7a4-8b833f66d378 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '28ab5ee2eeaf4e2a8644b087bdd21707'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:30.933016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1434/2846] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:30 2022] DELETE /identity/v3/users/28ab5ee2eeaf4e2a8644b087bdd21707 => generated 0 bytes in 303 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:31.321922 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:31.323415 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:31.342034 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03fdfde6-b807-45b5-803e-ff1520267b31 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:31.347799 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:31.404503 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:31.405137 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:31.405412 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:31.405741 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] PATH_INFO: `/v3/projects/506b78dcbadf409da9825aa26cd076bd` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:31.407603 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] RBAC: Authorizing `identity:delete_project(project_id=506b78dcbadf409da9825aa26cd076bd)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:31.411263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:31.520263 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '506b78dcbadf409da9825aa26cd076bd', 'request_id': 'req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:31.525635 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '506b78dcbadf409da9825aa26cd076bd', 'request_id': 'req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:31.600636 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '506b78dcbadf409da9825aa26cd076bd', 'request_id': 'req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:31.615569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] The token cache is being invalidate because project 506b78dcbadf409da9825aa26cd076bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:31.615860 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:31.616524 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-e0027fd3-ebc4-49c6-91f4-88dedfb6fe7c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:31.617993 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1413/2847] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:31 2022] DELETE /identity/v3/projects/506b78dcbadf409da9825aa26cd076bd => generated 0 bytes in 297 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:31.622969 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:31.623259 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:31.623540 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:31.646913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:31.648159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:31.691752 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:31.709019 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1435/2848] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:31.716893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:31.717178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:31.717452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:31.737745 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:31.738984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:31.746756 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1414/2849] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:31 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:31.751155 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:31.760608 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6648abc5-2801-4311-8762-7887c07b3397 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:31.765822 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:31.766433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:31.766703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:31.766992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:31.767624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:31.768203 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:31.769777 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:31.782520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1436/2850] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Sep 27 16:49:31.786752 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:31.796138 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03f4d0-1c7a-497f-bf90-ee0a1def3ef2 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:31.800853 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:31.801446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:31.801724 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:31.801998 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:31.802637 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:31.804574 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:31.904838 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1415/2851] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:31 2022] POST /identity/v3/projects => generated 401 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:31.909844 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:31.919857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f69c2577-2692-47f0-b14c-61c9c1f034a6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:31.975690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:31.976582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:31.976906 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:31.977220 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:31.978028 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:31.980179 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.066711 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1437/2852] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:31 2022] POST /identity/v3/users => generated 301 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:49:32.070984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.082437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c6b3962-0115-4b84-9338-ed335ae65941 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.087058 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.087665 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.088083 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.088357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.089031 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.090593 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.102480 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1416/2853] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:32 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:49:32.107344 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.118935 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1198d8a1-c62f-4e7f-8e73-b529a1dd2f6c admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.124016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585f4893-365e-4169-abb6-605605512611 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.125174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585f4893-365e-4169-abb6-605605512611 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.125474 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585f4893-365e-4169-abb6-605605512611 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.125890 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585f4893-365e-4169-abb6-605605512611 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.126545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585f4893-365e-4169-abb6-605605512611 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.128187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-585f4893-365e-4169-abb6-605605512611 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.139085 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1438/2854] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:32 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:49:32.144419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.155184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6014ac47-ba62-4400-93b6-453539d70148 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.160055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.160710 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.160990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.161262 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] PATH_INFO: `/v3/projects/bdf8a3302f0c4ccaaf86954f6695fc62/users/675a7b89704444e488290fb2d00aad9b/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.178454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] RBAC: Authorizing `identity:create_grant(project_id=bdf8a3302f0c4ccaaf86954f6695fc62, user_id=675a7b89704444e488290fb2d00aad9b, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.181883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.266930 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1417/2855] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:32 2022] PUT /identity/v3/projects/bdf8a3302f0c4ccaaf86954f6695fc62/users/675a7b89704444e488290fb2d00aad9b/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 123 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:32.278433 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.279327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.279327 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.306892 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] MFA Rules not processed for user `675a7b89704444e488290fb2d00aad9b`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:32.308064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:32.375448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1439/2856] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:32 2022] POST /identity/v3/auth/tokens => generated 3278 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:32.379889 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.414874 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:32.442158 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0b0567-d50b-49e5-b7c8-630438cd8062 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.446828 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.447432 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.447707 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.447974 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.448607 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.450557 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.584842 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1418/2857] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:32 2022] POST /identity/v3/projects => generated 401 bytes in 205 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:32.589310 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.600586 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5859a6e-6263-4a7b-8c68-06223fe319e1 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.642721 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:32.659832 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.660560 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.660705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.660974 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.661606 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.663602 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.804583 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1440/2858] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:32 2022] POST /identity/v3/users => generated 307 bytes in 216 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:32.808606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.823100 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5342b0f5-f38c-43cb-bad7-50e5aa5a22ef admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.828000 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.830906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.830906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.830906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.830906 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.831456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.843888 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1419/2859] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:32 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:32.847917 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.858207 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44e29833-2095-4917-b033-476d73be78b9 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:32.864187 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:32.864836 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:32.865184 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:32.865463 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] PATH_INFO: `/v3/projects/b1db894a618e4979b9c76a0404b0dd2c/users/04cc25401fbd4ceca2f1917ab6a8024d/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:32.882665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1db894a618e4979b9c76a0404b0dd2c, user_id=04cc25401fbd4ceca2f1917ab6a8024d, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:32.885477 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:32.977955 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1441/2860] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:32 2022] PUT /identity/v3/projects/b1db894a618e4979b9c76a0404b0dd2c/users/04cc25401fbd4ceca2f1917ab6a8024d/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:32.982913 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:32.995302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25ea502f-01bb-4edb-89e3-ca0915b964e0 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:33.066531 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71784321-0880-47f2-9bd1-d896dc218f4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:33.067174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71784321-0880-47f2-9bd1-d896dc218f4e admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:33.067483 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71784321-0880-47f2-9bd1-d896dc218f4e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:33.067762 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71784321-0880-47f2-9bd1-d896dc218f4e admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:33.068403 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71784321-0880-47f2-9bd1-d896dc218f4e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:33.070010 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71784321-0880-47f2-9bd1-d896dc218f4e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:33.080478 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1420/2861] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:32 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:49:33.084800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:33.096447 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d4eb185-2c08-4e03-9f93-c6dcb7df2020 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:33.101058 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:33.101965 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:33.102250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:33.102523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] PATH_INFO: `/v3/projects/b1db894a618e4979b9c76a0404b0dd2c/users/04cc25401fbd4ceca2f1917ab6a8024d/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:33.104502 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b1db894a618e4979b9c76a0404b0dd2c, user_id=04cc25401fbd4ceca2f1917ab6a8024d, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:33.107213 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:33.118545 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1442/2862] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:33 2022] PUT /identity/v3/projects/b1db894a618e4979b9c76a0404b0dd2c/users/04cc25401fbd4ceca2f1917ab6a8024d/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:33.129152 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:33.129437 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:33.129713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:33.162742 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] MFA Rules not processed for user `04cc25401fbd4ceca2f1917ab6a8024d`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:33.164695 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:33.253443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1421/2863] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:33 2022] POST /identity/v3/auth/tokens => generated 3345 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:33.266960 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:33.273851 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:33.363250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bec1deba-476d-431a-840a-461e39db07f6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:33.368166 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:33.368996 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:33.369379 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:33.369749 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:33.383205 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:33.385321 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:33.399374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1443/2864] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:33 2022] GET /identity/v3/auth/tokens => generated 3345 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:33.612814 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:33.623167 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-836c4190-50dc-4d0e-b161-e6608255b1fc None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:33.627425 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feda6fbd-625f-416f-b38d-9e54f0a421f1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:33.628119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda6fbd-625f-416f-b38d-9e54f0a421f1 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:33.628399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda6fbd-625f-416f-b38d-9e54f0a421f1 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:33.628677 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feda6fbd-625f-416f-b38d-9e54f0a421f1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:33.664323 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:33.695027 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feda6fbd-625f-416f-b38d-9e54f0a421f1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:33.696507 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feda6fbd-625f-416f-b38d-9e54f0a421f1 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:33.707110 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1422/2865] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:33 2022] GET /identity/v3/auth/tokens => generated 3278 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:38.830024 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:38.904452 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-478245f1-989b-4afd-9b00-3fa859003c0d service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:38.910856 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:38.911436 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:38.911713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:38.911984 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] PATH_INFO: `/v3/users/675a7b89704444e488290fb2d00aad9b` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:38.913134 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] RBAC: Authorizing `identity:delete_user(user_id=675a7b89704444e488290fb2d00aad9b)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:38.915125 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:39.114835 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:39.120607 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '675a7b89704444e488290fb2d00aad9b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:39.123562 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '675a7b89704444e488290fb2d00aad9b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:39.128247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '675a7b89704444e488290fb2d00aad9b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:39.128867 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '675a7b89704444e488290fb2d00aad9b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:39.269086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '675a7b89704444e488290fb2d00aad9b'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:39.287086 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1444/2866] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:38 2022] DELETE /identity/v3/users/675a7b89704444e488290fb2d00aad9b => generated 0 bytes in 458 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:39.291719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:39.292227 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:39.292341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:39.311928 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:39.313751 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:39.373647 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1423/2867] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:39 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:39.392398 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:39.394066 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:39.450605 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9602586c-4c9f-4d60-901d-2c409184acde admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:39.454412 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:39.455040 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:39.455313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:39.455578 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:39.464278 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:39.465690 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:39.474973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1445/2868] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:39 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:39.911212 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:39.912725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:39.926013 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fad5bef-8e38-4643-8a2a-994e215027fa None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:39.931080 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:39.931718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:39.931997 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:39.932283 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] PATH_INFO: `/v3/projects/bdf8a3302f0c4ccaaf86954f6695fc62` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:39.933452 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] RBAC: Authorizing `identity:delete_project(project_id=bdf8a3302f0c4ccaaf86954f6695fc62)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:39.935420 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:40.075482 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bdf8a3302f0c4ccaaf86954f6695fc62', 'request_id': 'req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.085882 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bdf8a3302f0c4ccaaf86954f6695fc62', 'request_id': 'req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.154174 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bdf8a3302f0c4ccaaf86954f6695fc62', 'request_id': 'req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.166575 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] The token cache is being invalidate because project bdf8a3302f0c4ccaaf86954f6695fc62 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:40.169458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.169458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.169458 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1424/2869] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:39 2022] DELETE /identity/v3/projects/bdf8a3302f0c4ccaaf86954f6695fc62 => generated 0 bytes in 258 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:40.173535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:40.183435 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10141561-a7c9-4f50-ae8d-8a638c0cce04 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:40.239669 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:40.240283 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:40.240559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:40.240825 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] PATH_INFO: `/v3/users/04cc25401fbd4ceca2f1917ab6a8024d` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:40.241925 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] RBAC: Authorizing `identity:delete_user(user_id=04cc25401fbd4ceca2f1917ab6a8024d)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:40.243875 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:40.487393 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04cc25401fbd4ceca2f1917ab6a8024d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.488072 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04cc25401fbd4ceca2f1917ab6a8024d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.494826 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04cc25401fbd4ceca2f1917ab6a8024d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.495465 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04cc25401fbd4ceca2f1917ab6a8024d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.499048 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:40.599992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-4dce583f-9eb6-449c-a493-ec9c02b28748 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04cc25401fbd4ceca2f1917ab6a8024d'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:40.618034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1446/2870] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:40 2022] DELETE /identity/v3/users/04cc25401fbd4ceca2f1917ab6a8024d => generated 0 bytes in 445 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:41.011395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.018835 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:41.028519 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c524f9f-a7bb-4967-97d6-ec49cf9ed97c admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.077471 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:41.087489 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.088163 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.088469 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.088744 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] PATH_INFO: `/v3/projects/b1db894a618e4979b9c76a0404b0dd2c` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.089959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] RBAC: Authorizing `identity:delete_project(project_id=b1db894a618e4979b9c76a0404b0dd2c)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.091893 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.215088 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b1db894a618e4979b9c76a0404b0dd2c', 'request_id': 'req-783a2905-5a20-4160-b4e1-af28c7ec163e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:41.222445 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b1db894a618e4979b9c76a0404b0dd2c', 'request_id': 'req-783a2905-5a20-4160-b4e1-af28c7ec163e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:41.300435 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b1db894a618e4979b9c76a0404b0dd2c', 'request_id': 'req-783a2905-5a20-4160-b4e1-af28c7ec163e'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:41.312362 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] The token cache is being invalidate because project b1db894a618e4979b9c76a0404b0dd2c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:41.312650 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:41.313277 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-783a2905-5a20-4160-b4e1-af28c7ec163e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:41.314757 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1425/2871] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:41 2022] DELETE /identity/v3/projects/b1db894a618e4979b9c76a0404b0dd2c => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:41.319775 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.322418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.322418 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.340576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:41.341787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:41.399707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1447/2872] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:41.409429 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.409713 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.409989 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.430104 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:41.431328 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:41.439141 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1426/2873] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:41 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:41.443275 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.453656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7470a0a-aa21-4ff6-8c84-c499444922e5 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.458466 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.459093 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.459371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.459639 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.460276 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:41.460575 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.462316 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.474286 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1448/2874] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:49:41.478310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.487652 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5caf4b47-25df-4c60-9436-1685f49994f4 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.492269 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.492851 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.493129 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.493399 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.494020 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.495965 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.576261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1427/2875] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:41 2022] POST /identity/v3/projects => generated 383 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:41.581751 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.591581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9eed0eee-71a6-46f9-a5d1-af47581b62ae admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.647397 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.647988 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.648265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.648538 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.649157 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.651092 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.729444 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1449/2876] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:41 2022] POST /identity/v3/users => generated 292 bytes in 148 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:41.733617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.743949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a6fdcdb-ebee-4d34-9dc2-ced78b47797e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.748627 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.752035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.752035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.752035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.752035 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.752466 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.761869 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1428/2877] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:41 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:41.766235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.776567 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd2766d6-34e3-4f17-84ba-7e2f67d03936 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.782067 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1feb13ea-03c4-44d2-ad54-a27ffd951e7a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.782746 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb13ea-03c4-44d2-ad54-a27ffd951e7a admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.783060 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb13ea-03c4-44d2-ad54-a27ffd951e7a admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.783331 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1feb13ea-03c4-44d2-ad54-a27ffd951e7a admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.783970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb13ea-03c4-44d2-ad54-a27ffd951e7a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.785570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1feb13ea-03c4-44d2-ad54-a27ffd951e7a admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.796083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1450/2878] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:41 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:41.800400 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:41.810313 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95b5c35-8047-4e19-8f15-abd7dcd5cf16 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:41.815357 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:41.815995 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.816275 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.816547 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] PATH_INFO: `/v3/projects/b8cda123e55b4e22ba965b605d12239d/users/963e0a2571754cc59d2c53bc4407c386/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.831805 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] RBAC: Authorizing `identity:create_grant(project_id=b8cda123e55b4e22ba965b605d12239d, user_id=963e0a2571754cc59d2c53bc4407c386, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:41.834518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:41.903055 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1429/2879] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:41 2022] PUT /identity/v3/projects/b8cda123e55b4e22ba965b605d12239d/users/963e0a2571754cc59d2c53bc4407c386/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 103 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:41.913284 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe062693-7615-4590-95e3-193fc3421894 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:41.913571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe062693-7615-4590-95e3-193fc3421894 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:41.913849 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe062693-7615-4590-95e3-193fc3421894 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:41.941624 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-fe062693-7615-4590-95e3-193fc3421894 None None] MFA Rules not processed for user `963e0a2571754cc59d2c53bc4407c386`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:41.942882 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-fe062693-7615-4590-95e3-193fc3421894 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:41.974633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:42.007543 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1451/2880] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:41 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:42.011954 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.074024 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd6a6300-9e97-44f8-b95f-1041ce41d2cf admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.079505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.080114 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.080390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.080658 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.081274 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.084166 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.230287 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1430/2881] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:42 2022] POST /identity/v3/projects => generated 385 bytes in 219 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:42.234118 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe062693-7615-4590-95e3-193fc3421894 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.245387 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe062693-7615-4590-95e3-193fc3421894 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.301367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.301968 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.302247 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.302520 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.303218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.305145 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.392236 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1452/2882] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:42 2022] POST /identity/v3/users => generated 299 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Sep 27 16:49:42.396719 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.402186 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:42.407750 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7561930-4a70-4bb3-8bd5-46be3233d62f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.412666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.413266 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.413544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.413817 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.414450 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.416085 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.426978 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1431/2883] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:42 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:49:42.431016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.441823 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-940dcaa3-b9f6-4b48-b705-f0202e2ea8af admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.446616 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.447318 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.447593 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.447888 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] PATH_INFO: `/v3/projects/ca429b7b92dc4f3e9401b883f8d531c0/users/d7675c1a8f954a0582d14cb35c7ba95c/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.462664 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca429b7b92dc4f3e9401b883f8d531c0, user_id=d7675c1a8f954a0582d14cb35c7ba95c, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.465332 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.531007 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1453/2884] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:42 2022] PUT /identity/v3/projects/ca429b7b92dc4f3e9401b883f8d531c0/users/d7675c1a8f954a0582d14cb35c7ba95c/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:42.535516 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.545564 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84b4ae56-c476-4e18-8526-4da2c152ccab admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.606122 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a1c9b8-cd25-4e27-97d2-3b1ba2b71ca4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.606722 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1c9b8-cd25-4e27-97d2-3b1ba2b71ca4 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.607025 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1c9b8-cd25-4e27-97d2-3b1ba2b71ca4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.607294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a1c9b8-cd25-4e27-97d2-3b1ba2b71ca4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.607918 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1c9b8-cd25-4e27-97d2-3b1ba2b71ca4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.609494 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a1c9b8-cd25-4e27-97d2-3b1ba2b71ca4 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.619638 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1432/2885] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:42 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:42.623807 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.633918 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-902dabcd-e671-48cf-827d-542e74a9bbda admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.638743 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.639406 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.639681 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.639950 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] PATH_INFO: `/v3/projects/ca429b7b92dc4f3e9401b883f8d531c0/users/d7675c1a8f954a0582d14cb35c7ba95c/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.641839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca429b7b92dc4f3e9401b883f8d531c0, user_id=d7675c1a8f954a0582d14cb35c7ba95c, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.644750 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.654928 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1454/2886] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:42 2022] PUT /identity/v3/projects/ca429b7b92dc4f3e9401b883f8d531c0/users/d7675c1a8f954a0582d14cb35c7ba95c/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:49:42.665562 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.665847 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.666121 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.694514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] MFA Rules not processed for user `d7675c1a8f954a0582d14cb35c7ba95c`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:42.695785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:42.762444 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1433/2887] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:42 2022] POST /identity/v3/auth/tokens => generated 3329 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:42.773313 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:42.775083 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:42.831388 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6b1d1c-23d0-4163-a438-fa11e4c0d408 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:42.838172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:42.838787 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:42.839076 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:42.839343 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:42.881051 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:42.897769 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:42.899335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:42.909615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1455/2888] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:42 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 137 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:48.109183 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:48.111521 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:48.123421 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76acbda9-e373-4e05-90f0-5f171e5b8a19 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:48.127512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21872b29-b9df-4a43-8979-f0554a2dfc46 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:48.128651 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21872b29-b9df-4a43-8979-f0554a2dfc46 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:48.128937 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21872b29-b9df-4a43-8979-f0554a2dfc46 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:48.129202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21872b29-b9df-4a43-8979-f0554a2dfc46 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:48.137990 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21872b29-b9df-4a43-8979-f0554a2dfc46 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:48.139446 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21872b29-b9df-4a43-8979-f0554a2dfc46 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:48.149682 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1434/2889] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:48 2022] GET /identity/v3/auth/tokens => generated 3329 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:52.939589 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=9 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:54.212442 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:54.273104 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44226d49-603c-42a7-8efd-3dc31e1d1f4c service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:54.277558 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:54.278168 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:54.278437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:54.278703 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] PATH_INFO: `/v3/users/963e0a2571754cc59d2c53bc4407c386` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:54.279830 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] RBAC: Authorizing `identity:delete_user(user_id=963e0a2571754cc59d2c53bc4407c386)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:54.281722 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:54.487292 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '963e0a2571754cc59d2c53bc4407c386'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:54.487874 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '963e0a2571754cc59d2c53bc4407c386'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:54.494650 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '963e0a2571754cc59d2c53bc4407c386'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:54.495367 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '963e0a2571754cc59d2c53bc4407c386'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:54.588177 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '963e0a2571754cc59d2c53bc4407c386'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:54.606661 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1456/2890] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:54 2022] DELETE /identity/v3/users/963e0a2571754cc59d2c53bc4407c386 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:54.611936 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:54.612426 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:54.612544 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:54.632109 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:54.633341 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:54.683123 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=100 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:54.691335 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1435/2891] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:54 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:54.701288 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:54.702705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:54.758816 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-644350ca-6daa-430a-8122-fa6fe0f554be admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:54.762297 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:54.762920 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:54.763198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:54.763468 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:54.772911 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:54.774333 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:54.784105 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1457/2892] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:54 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 83 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:49:55.199255 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:55.200821 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:55.213119 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b798cbd3-a43e-48a8-8753-80ee56a36af1 None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:55.217541 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:55.218159 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:55.218436 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:55.218708 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] PATH_INFO: `/v3/projects/b8cda123e55b4e22ba965b605d12239d` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:55.220791 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b8cda123e55b4e22ba965b605d12239d)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:55.222862 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:55.335716 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b8cda123e55b4e22ba965b605d12239d', 'request_id': 'req-5c46c2ee-b745-44e8-a02c-b278f87b2b91'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.341240 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b8cda123e55b4e22ba965b605d12239d', 'request_id': 'req-5c46c2ee-b745-44e8-a02c-b278f87b2b91'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.397395 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b8cda123e55b4e22ba965b605d12239d', 'request_id': 'req-5c46c2ee-b745-44e8-a02c-b278f87b2b91'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.409366 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] The token cache is being invalidate because project b8cda123e55b4e22ba965b605d12239d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:55.409701 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.410330 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.411822 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1436/2893] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:55 2022] DELETE /identity/v3/projects/b8cda123e55b4e22ba965b605d12239d => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:55.415871 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:55.425499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8203acd2-595a-497f-87db-f8138a5ff583 service neutron] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:55.444594 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:55.483737 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:55.484371 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:55.484649 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:55.484921 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] PATH_INFO: `/v3/users/d7675c1a8f954a0582d14cb35c7ba95c` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:55.486505 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] RBAC: Authorizing `identity:delete_user(user_id=d7675c1a8f954a0582d14cb35c7ba95c)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:55.488523 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:55.652035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'd7675c1a8f954a0582d14cb35c7ba95c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.652693 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'd7675c1a8f954a0582d14cb35c7ba95c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.659781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'd7675c1a8f954a0582d14cb35c7ba95c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.660413 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'd7675c1a8f954a0582d14cb35c7ba95c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.702035 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-eb3b9c00-baf4-46f3-9e23-0065704dbb6c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'd7675c1a8f954a0582d14cb35c7ba95c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:55.719415 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1458/2894] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:49:55 2022] DELETE /identity/v3/users/d7675c1a8f954a0582d14cb35c7ba95c => generated 0 bytes in 304 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:49:56.116294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.117785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:56.131845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c46c2ee-b745-44e8-a02c-b278f87b2b91 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.189079 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.189648 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.190016 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.190514 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] PATH_INFO: `/v3/projects/ca429b7b92dc4f3e9401b883f8d531c0` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.191604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca429b7b92dc4f3e9401b883f8d531c0)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.193505 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:56.289178 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca429b7b92dc4f3e9401b883f8d531c0', 'request_id': 'req-7819635d-6636-4801-811c-e7a2c526fc40'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:56.294679 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca429b7b92dc4f3e9401b883f8d531c0', 'request_id': 'req-7819635d-6636-4801-811c-e7a2c526fc40'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:56.377054 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca429b7b92dc4f3e9401b883f8d531c0', 'request_id': 'req-7819635d-6636-4801-811c-e7a2c526fc40'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:56.388883 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] The token cache is being invalidate because project ca429b7b92dc4f3e9401b883f8d531c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81613) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:49:56.394185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:56.394185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-7819635d-6636-4801-811c-e7a2c526fc40 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:49:56.394185 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1437/2895] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:49:56 2022] DELETE /identity/v3/projects/ca429b7b92dc4f3e9401b883f8d531c0 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:56.398052 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.398339 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.398615 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.419137 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:56.420414 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:56.478218 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1459/2896] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:56.486354 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.486643 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.486968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.506785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:56.507991 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:56.515726 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1438/2897] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:56 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:56.520570 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.530197 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48c6c44b-5261-4cd9-9915-775bda4699e4 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.535372 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.535956 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.536231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.536498 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] PATH_INFO: `/v3/domains` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.537138 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=81612) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Sep 27 16:49:56.537437 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.539055 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:56.550374 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1460/2898] 10.222.0.64 () {58 vars in 1153 bytes} [Tue Sep 27 16:49:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Sep 27 16:49:56.554543 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.563606 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e966001b-81b7-4d03-9328-831437c8f7ab None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.568352 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.568933 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.569209 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.569476 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.570099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.572014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:56.576413 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:56.673057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1439/2899] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:56 2022] POST /identity/v3/projects => generated 385 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:56.677232 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.687620 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-561ee3e7-7398-4d84-a78e-3aa54258d77e admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.713514 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:56.744336 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.744912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.745190 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.745461 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.746079 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.748081 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:56.837961 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1461/2900] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:56 2022] POST /identity/v3/users => generated 293 bytes in 161 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:56.842414 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.852932 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e922e5b3-6c02-407a-8c5e-ffce54bce671 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.857530 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.858124 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.858443 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.858709 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.859367 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.860949 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:56.870968 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1440/2901] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:56 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:56.875065 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.886726 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c812893d-abe4-48ff-930c-c05e87167866 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.892235 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b7ebcfd-1ffa-48e1-a442-e14c8290d627 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.892837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ebcfd-1ffa-48e1-a442-e14c8290d627 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.893781 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ebcfd-1ffa-48e1-a442-e14c8290d627 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.894057 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b7ebcfd-1ffa-48e1-a442-e14c8290d627 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.894688 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7ebcfd-1ffa-48e1-a442-e14c8290d627 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.896317 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b7ebcfd-1ffa-48e1-a442-e14c8290d627 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:56.906180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1462/2902] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:56 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:56.910549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:56.920556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00bf74b8-cf71-4c77-86b3-34694db3a50a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:56.925382 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:56.928935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:56.928935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:56.928935 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] PATH_INFO: `/v3/projects/e44204b12b9c48c1a5978b5fec9d0935/users/12017f3d7b9c4eec8b1602cd8fa932e5/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:56.941510 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] RBAC: Authorizing `identity:create_grant(project_id=e44204b12b9c48c1a5978b5fec9d0935, user_id=12017f3d7b9c4eec8b1602cd8fa932e5, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:56.944223 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.056988 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1441/2903] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:56 2022] PUT /identity/v3/projects/e44204b12b9c48c1a5978b5fec9d0935/users/12017f3d7b9c4eec8b1602cd8fa932e5/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 147 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:57.069848 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.070132 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.070404 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.097839 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] MFA Rules not processed for user `12017f3d7b9c4eec8b1602cd8fa932e5`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:57.104800 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:57.162010 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1463/2904] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:57 2022] POST /identity/v3/auth/tokens => generated 3262 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Sep 27 16:49:57.166646 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.227804 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-197f6a41-b771-4ff2-98a1-ac09a045c81a admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:57.232675 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:57.233280 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.233566 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.233844 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.234512 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:57.236456 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.367549 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1442/2905] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:57 2022] POST /identity/v3/projects => generated 383 bytes in 201 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:57.371705 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.382924 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9a237e3-4a4d-4bd2-8d90-30bc48495c61 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:57.439217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:57.440653 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.440942 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.441198 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.441805 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:57.443761 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.516912 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1464/2906] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:57 2022] POST /identity/v3/users => generated 292 bytes in 146 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:57.521190 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.531689 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fec9630b-e2ce-4169-a848-e82c1bd30ee1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:57.536856 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:57.537449 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.537725 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.537996 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.538631 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:57.540340 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.551818 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1443/2907] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:57.555733 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.565581 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8856cb6-91fe-405a-88d5-2cfb321d9dc6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:57.570100 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a995a98-11ab-4510-bf5d-0e2e42c61cc4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:57.570685 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a995a98-11ab-4510-bf5d-0e2e42c61cc4 admin admin] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.570992 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a995a98-11ab-4510-bf5d-0e2e42c61cc4 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.571265 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a995a98-11ab-4510-bf5d-0e2e42c61cc4 admin admin] PATH_INFO: `/v3/roles` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.571903 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a995a98-11ab-4510-bf5d-0e2e42c61cc4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:57.573469 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a995a98-11ab-4510-bf5d-0e2e42c61cc4 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.583857 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1465/2908] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:57.587803 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.598231 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-821a0b28-541a-4c26-96b9-085cc1d30711 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:57.602845 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:57.603486 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] REQUEST_METHOD: `PUT` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.603783 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.604057 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] PATH_INFO: `/v3/projects/6e7ae1b7a0634f749aecb354b363fbef/users/ebc6f0c3136b4ee7b5e1b98ea3a7d0ad/roles/1448af55b1904e1199e3954829e9815a` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.619642 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6e7ae1b7a0634f749aecb354b363fbef, user_id=ebc6f0c3136b4ee7b5e1b98ea3a7d0ad, role_id=1448af55b1904e1199e3954829e9815a)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:57.622293 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.679617 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1444/2909] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:57 2022] PUT /identity/v3/projects/6e7ae1b7a0634f749aecb354b363fbef/users/ebc6f0c3136b4ee7b5e1b98ea3a7d0ad/roles/1448af55b1904e1199e3954829e9815a => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Sep 27 16:49:57.690893 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.691174 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.691443 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.718675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] MFA Rules not processed for user `ebc6f0c3136b4ee7b5e1b98ea3a7d0ad`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:57.719909 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:57.724571 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:57.783370 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1466/2910] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:57 2022] POST /identity/v3/auth/tokens => generated 3260 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:57.788309 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.850666 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f0d6c91-0f0e-4ff4-afd4-c0967ed5d0a3 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:57.855310 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:57.855904 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:57.856184 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:57.856454 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] PATH_INFO: `/v3/projects` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:57.857099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:57.859032 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:57.864556 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:57.962270 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1445/2911] 10.222.0.64 () {60 vars in 1153 bytes} [Tue Sep 27 16:49:57 2022] POST /identity/v3/projects => generated 383 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:57.966128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:57.977483 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e430c39-af87-467f-97c9-28b56b788cc7 None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:58.031850 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:58.032439 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.032713 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.032979 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] PATH_INFO: `/v3/users` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.034214 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:58.036254 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:58.108348 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1467/2912] 10.222.0.64 () {60 vars in 1144 bytes} [Tue Sep 27 16:49:57 2022] POST /identity/v3/users => generated 298 bytes in 143 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Sep 27 16:49:58.112569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:58.122959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3faae6d-0ae3-40b1-82a1-26d2bc3f22ff admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:58.127984 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:58.128604 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.128886 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.129139 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.129770 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:58.131389 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:58.141290 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1446/2913] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Sep 27 16:49:58.146180 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:58.156172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cce325d3-e7ae-4aa9-8db1-84f5c13acc25 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:58.161667 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:58.162306 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.162584 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.162895 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] PATH_INFO: `/v3/projects/3379eccd0ca34772b1356c99aa73be66/users/e1996b63520541ceb8b31db282e283c2/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.177656 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3379eccd0ca34772b1356c99aa73be66, user_id=e1996b63520541ceb8b31db282e283c2, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:58.180341 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:58.261049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1468/2914] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:58 2022] PUT /identity/v3/projects/3379eccd0ca34772b1356c99aa73be66/users/e1996b63520541ceb8b31db282e283c2/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:58.265245 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:58.276775 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e0c1276-2a8a-4b5e-8502-38226eff5dec admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:58.333256 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c0aacaf-9e61-4c0f-942d-83c3df4921d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:58.333866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0aacaf-9e61-4c0f-942d-83c3df4921d9 admin admin] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.334150 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0aacaf-9e61-4c0f-942d-83c3df4921d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.334424 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c0aacaf-9e61-4c0f-942d-83c3df4921d9 admin admin] PATH_INFO: `/v3/roles` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.335099 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0aacaf-9e61-4c0f-942d-83c3df4921d9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:58.336694 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c0aacaf-9e61-4c0f-942d-83c3df4921d9 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:58.347712 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1447/2915] 10.222.0.64 () {58 vars in 1122 bytes} [Tue Sep 27 16:49:58 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Sep 27 16:49:58.351828 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:58.361973 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d446b970-90c5-4a3b-a8ca-7f36c6443646 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:58.366635 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:58.367302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] REQUEST_METHOD: `PUT` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.367582 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.367854 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] PATH_INFO: `/v3/projects/3379eccd0ca34772b1356c99aa73be66/users/e1996b63520541ceb8b31db282e283c2/roles/bd6f5e4de7d6438c962970b4fda1866a` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.369742 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] RBAC: Authorizing `identity:create_grant(project_id=3379eccd0ca34772b1356c99aa73be66, user_id=e1996b63520541ceb8b31db282e283c2, role_id=bd6f5e4de7d6438c962970b4fda1866a)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:58.372427 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:58.383124 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1469/2916] 10.222.0.64 () {60 vars in 1483 bytes} [Tue Sep 27 16:49:58 2022] PUT /identity/v3/projects/3379eccd0ca34772b1356c99aa73be66/users/e1996b63520541ceb8b31db282e283c2/roles/bd6f5e4de7d6438c962970b4fda1866a => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:49:58.393994 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] REQUEST_METHOD: `POST` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.394294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.394569 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.422014 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.auth.core [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] MFA Rules not processed for user `e1996b63520541ceb8b31db282e283c2`. Rule list: `[]` (Enabled: `True`). {{(pid=81613) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:49:58.423253 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:58.489551 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1448/2917] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:49:58 2022] POST /identity/v3/auth/tokens => generated 3327 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:49:58.501250 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:49:58.503599 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:49:58.561231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-119f8f5c-0d46-43ba-acb2-37f7882a669d admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:49:58.565376 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:49:58.565970 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:49:58.566238 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:49:58.566504 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:49:58.575553 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:49:58.622217 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:49:58.623675 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:49:58.633493 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1470/2918] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:49:58 2022] GET /identity/v3/auth/tokens => generated 3262 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:50:00.536950 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:00.546464 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:00.551319 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6aae755-3a0b-4aa8-bd03-b4ab275c598a None None] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:00.555067 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:00.555668 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:00.555948 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:00.556220 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:00.608197 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:00.611829 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:00.613982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:00.623222 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1449/2919] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:50:00 2022] GET /identity/v3/auth/tokens => generated 3260 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:50:02.035448 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:02.044016 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda5a7cd-bbd9-46dc-b8cc-2301fa956235 service cinder] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:02.047633 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c793c623-5527-46cd-a4dc-927fd546dc0a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f8babdc682094c18a4ba710224815a27', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:02.048221 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c793c623-5527-46cd-a4dc-927fd546dc0a service cinder] REQUEST_METHOD: `GET` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:02.048497 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c793c623-5527-46cd-a4dc-927fd546dc0a service cinder] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:02.048768 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c793c623-5527-46cd-a4dc-927fd546dc0a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:02.057716 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c793c623-5527-46cd-a4dc-927fd546dc0a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:02.059165 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c793c623-5527-46cd-a4dc-927fd546dc0a service cinder] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:02.069159 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1471/2920] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:50:02 2022] GET /identity/v3/auth/tokens => generated 3327 bytes in 34 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:50:06.640861 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:06.702788 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e191337-9ae8-4629-b8fc-7eca821550e0 service cinder] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:06.707302 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:06.707924 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:06.708202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:06.708509 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] PATH_INFO: `/v3/users/12017f3d7b9c4eec8b1602cd8fa932e5` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:06.709785 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] RBAC: Authorizing `identity:delete_user(user_id=12017f3d7b9c4eec8b1602cd8fa932e5)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:06.711738 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:06.905050 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '12017f3d7b9c4eec8b1602cd8fa932e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:06.912558 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '12017f3d7b9c4eec8b1602cd8fa932e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:06.913261 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '12017f3d7b9c4eec8b1602cd8fa932e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:06.913859 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '12017f3d7b9c4eec8b1602cd8fa932e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.014343 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '12017f3d7b9c4eec8b1602cd8fa932e5'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.035062 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1450/2921] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:50:06 2022] DELETE /identity/v3/users/12017f3d7b9c4eec8b1602cd8fa932e5 => generated 0 bytes in 395 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:50:07.040298 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] REQUEST_METHOD: `POST` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:07.040803 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:07.040913 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:07.060559 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.auth.core [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] MFA Rules not processed for user `3c05ec42fc16441e83ec77e10313ea85`. Rule list: `[]` (Enabled: `True`). {{(pid=81612) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Sep 27 16:50:07.065204 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:07.121731 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1472/2922] 10.222.0.64 () {58 vars in 958 bytes} [Tue Sep 27 16:50:07 2022] POST /identity/v3/auth/tokens => generated 3239 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Sep 27 16:50:07.131518 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:07.132972 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:07.189866 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-244886e0-50f3-4604-a89f-7aeaa748560f admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:07.193908 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '28f0f9fef2f54e56a7a92d077c25d2b7', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:07.194502 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:07.194787 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:07.195074 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:07.203855 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:07.205316 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:07.215754 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1451/2923] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:50:07 2022] GET /identity/v3/auth/tokens => generated 3239 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:50:07.616352 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:07.617901 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:07.633206 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-183d0de3-47af-42c2-a86a-6de89c53dbba None None] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:07.638147 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:07.638985 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:07.639267 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:07.639630 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] PATH_INFO: `/v3/projects/e44204b12b9c48c1a5978b5fec9d0935` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:07.640785 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] RBAC: Authorizing `identity:delete_project(project_id=e44204b12b9c48c1a5978b5fec9d0935)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:07.645467 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:07.742293 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e44204b12b9c48c1a5978b5fec9d0935', 'request_id': 'req-04c516c3-f4e4-445f-b8e8-c491dca7c0be'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.804784 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e44204b12b9c48c1a5978b5fec9d0935', 'request_id': 'req-04c516c3-f4e4-445f-b8e8-c491dca7c0be'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.811231 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e44204b12b9c48c1a5978b5fec9d0935', 'request_id': 'req-04c516c3-f4e4-445f-b8e8-c491dca7c0be'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.822244 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] The token cache is being invalidate because project e44204b12b9c48c1a5978b5fec9d0935 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:50:07.822535 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.823186 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:07.824670 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1473/2924] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:50:07 2022] DELETE /identity/v3/projects/e44204b12b9c48c1a5978b5fec9d0935 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:50:07.828740 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:07.839304 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12e0f422-f975-43e8-8a8c-6db2255c3173 service neutron] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:07.850801 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:07.896718 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:07.897347 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:07.897622 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:07.897892 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] PATH_INFO: `/v3/users/ebc6f0c3136b4ee7b5e1b98ea3a7d0ad` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:07.899021 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ebc6f0c3136b4ee7b5e1b98ea3a7d0ad)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:07.901008 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:08.103199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ebc6f0c3136b4ee7b5e1b98ea3a7d0ad'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.110132 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ebc6f0c3136b4ee7b5e1b98ea3a7d0ad'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.110813 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ebc6f0c3136b4ee7b5e1b98ea3a7d0ad'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.111419 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ebc6f0c3136b4ee7b5e1b98ea3a7d0ad'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.208714 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ebc6f0c3136b4ee7b5e1b98ea3a7d0ad'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.227294 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1452/2925] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:50:07 2022] DELETE /identity/v3/users/ebc6f0c3136b4ee7b5e1b98ea3a7d0ad => generated 0 bytes in 399 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:50:08.606612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:08.606612 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:08.620172 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04c516c3-f4e4-445f-b8e8-c491dca7c0be admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:08.638866 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:08.678335 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:08.679020 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:08.679302 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:08.679576 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] PATH_INFO: `/v3/projects/6e7ae1b7a0634f749aecb354b363fbef` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:08.680694 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6e7ae1b7a0634f749aecb354b363fbef)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:08.682686 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:08.781837 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6e7ae1b7a0634f749aecb354b363fbef', 'request_id': 'req-7958ba35-b8fd-437d-a426-e1854fff5ba6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.838665 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6e7ae1b7a0634f749aecb354b363fbef', 'request_id': 'req-7958ba35-b8fd-437d-a426-e1854fff5ba6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.845153 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6e7ae1b7a0634f749aecb354b363fbef', 'request_id': 'req-7958ba35-b8fd-437d-a426-e1854fff5ba6'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.856049 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] The token cache is being invalidate because project 6e7ae1b7a0634f749aecb354b363fbef was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:50:08.856334 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.857006 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:08.858499 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1474/2926] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:50:08 2022] DELETE /identity/v3/projects/6e7ae1b7a0634f749aecb354b363fbef => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:50:08.862959 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:08.864356 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:08.877692 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6ea1f2d-2050-4c19-a01e-41af844f97f1 admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:08.934199 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:08.934887 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] REQUEST_METHOD: `DELETE` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:08.935172 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:08.935475 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] PATH_INFO: `/v3/users/e1996b63520541ceb8b31db282e283c2` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:08.936594 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] RBAC: Authorizing `identity:delete_user(user_id=e1996b63520541ceb8b31db282e283c2)` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:08.938554 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:09.107271 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=84,keystone:DELETE=15,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:09.115390 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e1996b63520541ceb8b31db282e283c2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.122863 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e1996b63520541ceb8b31db282e283c2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.123513 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e1996b63520541ceb8b31db282e283c2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.124111 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e1996b63520541ceb8b31db282e283c2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.199075 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.notifications [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e1996b63520541ceb8b31db282e283c2'} {{(pid=81613) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.217350 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1453/2927] 10.222.0.64 () {58 vars in 1224 bytes} [Tue Sep 27 16:50:08 2022] DELETE /identity/v3/users/e1996b63520541ceb8b31db282e283c2 => generated 0 bytes in 355 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:50:09.634834 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Authenticating user token {{(pid=81612) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:09.636707 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.fernet_utils [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81612) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:09.652034 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7958ba35-b8fd-437d-a426-e1854fff5ba6 admin admin] Validating token access rules against request {{(pid=81612) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:09.711489 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '3c05ec42fc16441e83ec77e10313ea85', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '9ac75ab2f7e84b088f06cdcdbcfae272', 'project_domain_id': 'default', 'roles': ['reader', 'admin', 'member'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81612) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:09.712120 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] REQUEST_METHOD: `DELETE` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:09.712395 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] SCRIPT_NAME: `/identity` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:09.712666 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] PATH_INFO: `/v3/projects/3379eccd0ca34772b1356c99aa73be66` {{(pid=81612) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:09.713770 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3379eccd0ca34772b1356c99aa73be66)` {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:09.715729 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] RBAC: Authorization granted {{(pid=81612) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:09.885194 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3379eccd0ca34772b1356c99aa73be66', 'request_id': 'req-44fb2b84-20a2-452c-8d48-326faf8c467c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.888995 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:09.964495 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3379eccd0ca34772b1356c99aa73be66', 'request_id': 'req-44fb2b84-20a2-452c-8d48-326faf8c467c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.971763 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3379eccd0ca34772b1356c99aa73be66', 'request_id': 'req-44fb2b84-20a2-452c-8d48-326faf8c467c'} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.983128 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] The token cache is being invalidate because project 3379eccd0ca34772b1356c99aa73be66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=81612) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Sep 27 16:50:09.983416 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.984064 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG keystone.notifications [None req-44fb2b84-20a2-452c-8d48-326faf8c467c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=81612) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Sep 27 16:50:09.985536 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: [pid: 81612|app: 0|req: 1475/2928] 10.222.0.64 () {58 vars in 1233 bytes} [Tue Sep 27 16:50:09 2022] DELETE /identity/v3/projects/3379eccd0ca34772b1356c99aa73be66 => generated 0 bytes in 351 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Sep 27 16:50:19.273982 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=15,keystone:DELETE=5,keystone:INSERT=1 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:20.088863 dv-cinder-855060-2-iscsi devstack@keystone.service[81612]: DEBUG dbcounter [-] [81612] Writing DB stats keystone:SELECT=6,keystone:DELETE=2 {{(pid=81612) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Sep 27 16:50:44.365202 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Authenticating user token {{(pid=81613) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Sep 27 16:50:44.366623 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:44.437599 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2988abe8-d3a4-4430-b4d8-a09596cf4e5e admin admin] Validating token access rules against request {{(pid=81613) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Sep 27 16:50:44.495324 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '4a39bfa348af4f499621fcb5d612e82c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '6519177f56a241148d3389f111fb8b28', 'project_domain_id': 'default', 'roles': ['service', 'admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=81613) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Sep 27 16:50:44.495925 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] REQUEST_METHOD: `GET` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Sep 27 16:50:44.496204 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] SCRIPT_NAME: `/identity` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Sep 27 16:50:44.496473 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=81613) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Sep 27 16:50:44.498182 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.fernet_utils [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=81613) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Sep 27 16:50:44.509614 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Sep 27 16:50:44.511371 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbdee378-b914-42e0-91d4-39621f2354d1 service placement] RBAC: Authorization granted {{(pid=81613) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Sep 27 16:50:44.521305 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: [pid: 81613|app: 0|req: 1454/2929] 10.222.0.64 () {60 vars in 1365 bytes} [Tue Sep 27 16:50:44 2022] GET /identity/v3/auth/tokens => generated 3308 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Sep 27 16:50:54.517146 dv-cinder-855060-2-iscsi devstack@keystone.service[81613]: DEBUG dbcounter [-] [81613] Writing DB stats keystone:SELECT=58 {{(pid=81613) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}